2024-03-29T03:58:27.130 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T03:58:27.131 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080 --name mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi --owner scheduled_mchangir@teuthology --description fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining} -- /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/orig.config.yaml 2024-03-29T03:58:27.178 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T03:58:27.271 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080 branch: wip-mchangir-testing1-main-20240328-072412 description: fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining} email: mchangir@redhat.com first_in_suite: false job_id: '7629080' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing1-main-20240328-072412 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 69e2cd73c52284dfd5096c66b60c6e6b26a3c447 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 69e2cd73c52284dfd5096c66b60c6e6b26a3c447 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing1-main-20240328-072412 sha1: 69e2cd73c52284dfd5096c66b60c6e6b26a3c447 owner: scheduled_mchangir@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8207 sha1: 69e2cd73c52284dfd5096c66b60c6e6b26a3c447 sleep_before_teardown: 0 suite: fs:functional suite_branch: wip-mchangir-testing1-main-20240328-072412 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 69e2cd73c52284dfd5096c66b60c6e6b26a3c447 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvoTFsWWL4vZ7GzTmlpqlvDFCSFJM8MPUM/4cPygSQpM4a+eE5+Ilqcdje7rDFxqSiEeHkRd0vNIKMSl6fIEck= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2iheM/phZGpxb/nWh+qNqJPh21pUeEuzfV1bo+S1e/XORiFLwi+n9nvbrRFTnjMjUmEIh7WVJLrqjXkilT/Nk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_uninlining teuthology: fragments_dropped: - /cephfs/home/mchangir/src/git.ceph.com_ceph-c_wip-mchangir-testing1-main-20240328-072412/qa/suites/fs/functional/overrides/pg_health.yaml meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_02:01:30 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T03:58:27.271 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa; will attempt to use it 2024-03-29T03:58:27.272 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks 2024-03-29T03:58:27.272 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T03:58:27.273 INFO:teuthology.task.internal:Checking packages... 2024-03-29T03:58:27.300 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '69e2cd73c52284dfd5096c66b60c6e6b26a3c447' 2024-03-29T03:58:27.300 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T03:58:27.300 INFO:teuthology.packaging:ref: None 2024-03-29T03:58:27.300 INFO:teuthology.packaging:tag: None 2024-03-29T03:58:27.301 INFO:teuthology.packaging:branch: wip-mchangir-testing1-main-20240328-072412 2024-03-29T03:58:27.301 INFO:teuthology.packaging:sha1: 69e2cd73c52284dfd5096c66b60c6e6b26a3c447 2024-03-29T03:58:27.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing1-main-20240328-072412 2024-03-29T03:58:27.447 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2520.g69e2cd73 2024-03-29T03:58:27.449 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T03:58:27.455 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T03:58:27.455 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T03:58:27.496 INFO:teuthology.task.internal:Saving configuration 2024-03-29T03:58:27.513 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T03:58:27.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T03:58:27.544 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 03:53:10.013994', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvoTFsWWL4vZ7GzTmlpqlvDFCSFJM8MPUM/4cPygSQpM4a+eE5+Ilqcdje7rDFxqSiEeHkRd0vNIKMSl6fIEck='} 2024-03-29T03:58:27.564 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 03:53:10.012152', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2iheM/phZGpxb/nWh+qNqJPh21pUeEuzfV1bo+S1e/XORiFLwi+n9nvbrRFTnjMjUmEIh7WVJLrqjXkilT/Nk='} 2024-03-29T03:58:27.564 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T03:58:27.572 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-29T03:58:27.572 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-29T03:58:27.572 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T03:58:27.662 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f365a22af70>, signals=[15]) 2024-03-29T03:58:27.663 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T03:58:27.670 INFO:teuthology.task.internal:Opening connections... 2024-03-29T03:58:27.670 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T03:58:27.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T03:58:27.766 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T03:58:27.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T03:58:27.855 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T03:58:27.862 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-03-29T03:58:27.879 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-03-29T03:58:27.880 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-03-29T03:58:27.937 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T03:58:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T03:58:27.939 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T03:58:27.939 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-03-29T03:58:27.959 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-29T03:58:27.976 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-29T03:58:27.976 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-29T03:58:28.031 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T03:58:28.032 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T03:58:28.033 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-29T03:58:28.059 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T03:58:28.069 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T03:58:28.087 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T03:58:28.087 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-03-29T03:58:28.091 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-29T03:58:28.106 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T03:58:28.114 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T03:58:28.114 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-03-29T03:58:28.147 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-29T03:58:28.379 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T03:58:28.441 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T03:58:28.456 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T03:58:28.456 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T03:58:28.457 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T03:58:28.457 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T03:58:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T03:58:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T03:58:28.475 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-03-29T03:58:28.475 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-29T03:58:28.530 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-03-29T03:58:28.530 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-03-29T03:58:28.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-03-29T03:58:28.535 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-361.el9.x86_64 2024-03-29T03:58:28.535 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-361.el9.x86_64 2024-03-29T03:58:28.535 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-03-29T03:58:30.747 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T03:58:30.883 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T03:58:32.455 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-29T03:58:32.517 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-29T03:58:33.354 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T03:58:34.765 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-03-29T03:58:34.980 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-03-29T03:58:37.566 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-29T03:58:39.089 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T03:58:40.565 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-29T03:58:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T03:58:42.661 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-29T03:58:44.934 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-29T03:58:46.564 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T03:58:47.233 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T03:58:49.531 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T03:58:49.532 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T03:58:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-29T03:58:49.656 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-29T03:58:49.656 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-29T03:58:49.656 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-29T03:58:49.656 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:Install 4 Packages 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 76 M 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 123 M 2024-03-29T03:58:49.657 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-29T03:58:49.893 INFO:teuthology.orchestra.run.smithi156.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 815 kB/s | 192 kB 00:00 2024-03-29T03:58:50.568 INFO:teuthology.orchestra.run.smithi156.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-29T03:58:51.493 INFO:teuthology.orchestra.run.smithi156.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 18 MB/s | 34 MB 00:01 2024-03-29T03:58:51.562 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-29T03:58:51.743 INFO:teuthology.orchestra.run.smithi156.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T03:58:51.747 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-29T03:58:51.747 INFO:teuthology.orchestra.run.smithi156.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-29T03:58:51.747 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-29T03:58:52.124 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-29T03:58:52.124 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-29T03:58:52.947 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-29T03:58:52.948 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-29T03:58:53.778 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T03:58:53.778 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T03:58:53.867 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T03:58:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 76 M 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-03-29T03:58:53.881 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T03:58:54.123 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 795 kB/s | 192 kB 00:00 2024-03-29T03:58:55.148 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-03-29T03:58:55.250 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-29T03:58:55.607 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-03-29T03:58:55.851 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T03:58:55.952 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T03:58:56.306 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 13 MB/s | 28 MB 00:02 2024-03-29T03:58:56.310 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T03:58:56.310 INFO:teuthology.orchestra.run.smithi203.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-29T03:58:56.310 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T03:58:56.686 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T03:58:56.686 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T03:58:57.539 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T03:58:57.539 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T03:58:58.717 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T03:58:58.857 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T03:58:59.861 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T03:59:00.486 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T03:59:00.567 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T03:59:03.368 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T03:59:03.506 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T03:59:03.600 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T03:59:03.713 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:08.219 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T03:59:08.263 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:08.447 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:12.985 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:37.617 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:37.617 INFO:teuthology.orchestra.run.smithi156.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T03:59:37.618 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:59:37.660 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:38.000 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:38.000 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T03:59:38.000 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T03:59:38.000 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:59:38.400 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-29T03:59:38.736 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T03:59:38.852 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T03:59:38.852 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T03:59:38.852 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T03:59:38.852 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T03:59:38.853 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T03:59:38.853 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T03:59:38.853 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T03:59:38.853 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T03:59:38.934 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T03:59:38.934 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T03:59:38.934 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2024-03-29T03:59:38.968 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2024-03-29T03:59:38.968 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2024-03-29T03:59:38.968 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.06 2024-03-29T03:59:38.968 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 68.el9 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Group : Unspecified 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 8264254 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : builder@centos.org 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T03:59:38.969 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T03:59:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T03:59:38.971 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T03:59:38.971 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 5.14.0-434.el9.x86_64 2024-03-29T03:59:38.971 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T03:59:39.582 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2024-03-29T03:59:40.956 INFO:teuthology.orchestra.run.smithi156.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T03:59:40.982 INFO:teuthology.orchestra.run.smithi156.stderr:done 2024-03-29T03:59:40.984 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-29T03:59:40.999 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.a33xevqQ70 2024-03-29T03:59:40.999 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.a33xevqQ70 2024-03-29T03:59:41.076 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.a33xevqQ70 2024-03-29T03:59:41.221 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.a33xevqQ70 is 6KB 2024-03-29T03:59:41.274 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.a33xevqQ70 2024-03-29T03:59:41.288 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T03:59:41.353 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T03:59:41.353 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T03:59:41.353 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T03:59:41.353 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T03:59:41.354 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T03:59:41.479 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2024-03-29T03:59:41.559 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-29T03:59:42.577 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:42.577 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T03:59:42.577 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:59:42.620 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:42.988 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:42.988 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T03:59:42.988 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T03:59:42.989 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T03:59:43.420 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:59:43.421 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T03:59:43.770 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T03:59:43.866 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T03:59:43.866 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T03:59:43.866 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T03:59:43.866 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T03:59:43.866 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T03:59:43.866 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T03:59:43.867 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T03:59:43.867 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T03:59:43.946 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T03:59:43.947 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T03:59:43.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-03-29T03:59:43.981 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T03:59:43.982 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T03:59:43.983 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T03:59:43.984 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T03:59:43.984 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-434.el9.x86_64 2024-03-29T03:59:43.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T03:59:44.579 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-03-29T03:59:46.002 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T03:59:46.040 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-03-29T03:59:46.043 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-29T03:59:46.058 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.VxmEtaysg2 2024-03-29T03:59:46.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VxmEtaysg2 2024-03-29T03:59:46.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.VxmEtaysg2 2024-03-29T03:59:46.263 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.VxmEtaysg2 is 6KB 2024-03-29T03:59:46.311 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.VxmEtaysg2 2024-03-29T03:59:46.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T03:59:46.390 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T03:59:46.390 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T03:59:46.390 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T03:59:46.390 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T03:59:46.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T03:59:46.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-03-29T04:00:11.508 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T04:00:11.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:00:11.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:00:16.538 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T04:00:16.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T04:00:16.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:00:46.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-29T04:00:51.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-29T04:00:55.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:00:55.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:01:00.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T04:01:00.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:01:13.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-29T04:01:18.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-29T04:01:25.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:01:25.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:01:28.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-29T04:01:30.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T04:01:30.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:01:31.985 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-29T04:01:32.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T04:01:32.475 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T04:01:32.475 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-29T04:01:32.491 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-434.el9.x86_64 2024-03-29T04:01:32.491 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T04:01:32.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T04:01:32.491 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T04:01:33.492 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T04:01:33.492 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T04:01:33.547 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-03-29T04:01:33.600 DEBUG:teuthology.parallel:result is None 2024-03-29T04:01:43.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:01:43.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:01:44.079 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-29T04:01:44.467 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:01:44.468 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T04:01:44.468 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-03-29T04:01:44.483 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-434.el9.x86_64 2024-03-29T04:01:44.483 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T04:01:44.483 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T04:01:44.483 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T04:01:45.484 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T04:01:45.484 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T04:01:45.563 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-03-29T04:01:45.605 DEBUG:teuthology.parallel:result is None 2024-03-29T04:01:45.606 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T04:01:45.613 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T04:01:45.613 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T04:01:45.616 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T04:01:45.637 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T04:01:45.657 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T04:01:45.691 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T04:01:45.692 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T04:01:45.695 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T04:01:45.829 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T04:01:45.836 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T04:01:45.836 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T04:01:45.839 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T04:01:45.874 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T04:01:45.877 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T04:01:45.888 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T04:01:45.891 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T04:01:45.893 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T04:01:45.900 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T04:01:45.900 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T04:01:45.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T04:01:45.967 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T04:01:45.977 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T04:01:45.977 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T04:01:46.005 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T04:01:46.022 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T04:01:46.096 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T04:01:46.150 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T04:01:46.229 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T04:01:46.275 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:01:46.275 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T04:01:46.343 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T04:01:46.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T04:01:46.441 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T04:01:46.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T04:01:46.570 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:01:46.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T04:01:46.639 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-03-29T04:01:46.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-29T04:01:46.686 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T04:01:46.742 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T04:01:47.190 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T04:01:47.197 INFO:teuthology.task.internal:Starting timer... 2024-03-29T04:01:47.198 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T04:01:47.236 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T04:01:47.268 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-29T04:01:47.269 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-03-29T04:01:47.330 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-03-29T04:01:47.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-03-29T04:01:47.389 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-03-29T04:01:47.391 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T04:01:47.392 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-03-29T04:01:47.440 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-03-29T04:01:47.440 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-03-29T04:01:47.480 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-03-29T04:01:47.480 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi156.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-03-29T04:01:47.480 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T04:01:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:01:47.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T04:01:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:01:47.549 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T04:01:47.549 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-03-29T04:01:47.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-03-29T04:01:47.656 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T04:01:47.665 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T04:01:47.681 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T04:01:47.682 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi156.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T04:06:01.830 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-29T04:06:01.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:06:01.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:06:01.925 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-29T04:06:02.004 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-29T04:06:02.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T04:06:02.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T04:06:02.093 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-29T04:06:02.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T04:06:02.170 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T04:06:02.180 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T04:06:02.180 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T04:06:02.181 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T04:06:02.183 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T04:06:02.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T04:06:02.212 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T04:06:02.229 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T04:06:02.240 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T04:06:02.257 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T04:06:02.261 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-03-29T04:06:02.277 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-03-29T04:06:02.287 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-03-29T04:06:02.295 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T04:06:02.300 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-03-29T04:06:02.311 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T04:06:02.317 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T04:06:02.333 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T04:06:02.370 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-29T04:06:02.373 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T04:06:02.373 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-29T04:06:02.373 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.373 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.374 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.374 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.383 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-29T04:06:02.387 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T04:06:02.387 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-29T04:06:02.387 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.388 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.388 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.388 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:06:02.388 INFO:teuthology.run_tasks:Running task install... 2024-03-29T04:06:02.396 DEBUG:teuthology.task.install:project ceph 2024-03-29T04:06:02.396 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447'}} 2024-03-29T04:06:02.396 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447'} 2024-03-29T04:06:02.397 INFO:teuthology.task.install:Using flavor: default 2024-03-29T04:06:02.402 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T04:06:02.402 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-29T04:06:02.403 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447', 'tag': None, 'wait_for_package': False} 2024-03-29T04:06:02.403 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=69e2cd73c52284dfd5096c66b60c6e6b26a3c447 2024-03-29T04:06:02.405 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447', 'tag': None, 'wait_for_package': False} 2024-03-29T04:06:02.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=69e2cd73c52284dfd5096c66b60c6e6b26a3c447 2024-03-29T04:06:02.550 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/ 2024-03-29T04:06:02.550 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2520.g69e2cd73 2024-03-29T04:06:02.563 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/ 2024-03-29T04:06:02.563 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2520.g69e2cd73 2024-03-29T04:06:02.721 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T04:06:02.721 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:06:02.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T04:06:02.731 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mchangir-testing1-main-20240328-072412/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T04:06:02.732 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:06:02.732 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T04:06:02.749 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-29T04:06:02.750 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T04:06:02.760 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-29T04:06:02.760 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/69e2cd73c52284dfd5096c66b60c6e6b26a3c447/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T04:06:02.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T04:06:02.827 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T04:06:02.898 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T04:06:02.907 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T04:06:02.964 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-03-29T04:06:02.965 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-03-29T04:06:02.974 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-03-29T04:06:02.976 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-03-29T04:06:03.203 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-03-29T04:06:03.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-29T04:06:03.231 INFO:teuthology.orchestra.run.smithi156.stdout:74 files removed 2024-03-29T04:06:03.257 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-29T04:06:03.863 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 252 kB/s | 83 kB 00:00 2024-03-29T04:06:03.957 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 208 kB/s | 83 kB 00:00 2024-03-29T04:06:04.256 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-03-29T04:06:04.391 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-03-29T04:06:04.598 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-29T04:06:04.767 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-29T04:06:05.017 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T04:06:05.186 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T04:06:06.642 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T04:06:06.661 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-29T04:06:08.224 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-03-29T04:06:08.435 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-03-29T04:06:13.886 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-29T04:06:14.167 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 19 MB 00:04 2024-03-29T04:06:17.961 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T04:06:18.361 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-29T04:06:20.052 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-29T04:06:20.445 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-29T04:06:21.963 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T04:06:22.163 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T04:06:26.844 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T04:06:27.196 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-29T04:06:29.411 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-29T04:06:29.412 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-29T04:06:29.423 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-29T04:06:29.424 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-29T04:06:29.425 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-29T04:06:29.426 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-29T04:06:29.428 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-29T04:06:29.430 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-29T04:06:29.431 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-29T04:06:29.510 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 6.6 k 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 5.4 M 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 890 k 2024-03-29T04:06:29.523 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 145 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 1.5 M 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 141 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 3.5 M 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 7.4 M 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 49 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 9.8 M 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 48 M 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 264 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 762 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 221 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 27 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 18 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-29T04:06:29.524 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 32 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 746 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 127 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 162 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 321 k 2024-03-29T04:06:29.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 298 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 99 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 85 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 3.1 M 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 170 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-29T04:06:29.526 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 3.4 M 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 3.1 M 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-29T04:06:29.527 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 21 M 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 34 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 2.4 M 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 249 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 4.7 M 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 17 M 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 16 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 25 k 2024-03-29T04:06:29.528 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 163 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T04:06:29.529 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 500 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 5.1 M 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T04:06:29.530 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 45 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 134 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T04:06:29.531 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T04:06:29.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T04:06:29.533 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T04:06:29.534 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T04:06:29.535 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout:Install 150 Packages 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 24 Packages 2024-03-29T04:06:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:06:29.537 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 226 M 2024-03-29T04:06:29.537 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T04:06:29.772 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-29T04:06:29.773 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-29T04:06:29.788 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-29T04:06:29.789 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-29T04:06:29.790 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-29T04:06:29.792 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-29T04:06:29.793 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-29T04:06:29.796 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-29T04:06:29.797 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-29T04:06:29.878 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 6.6 k 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 5.4 M 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 890 k 2024-03-29T04:06:29.890 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 145 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 1.5 M 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 141 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 3.5 M 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 7.4 M 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 49 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 9.8 M 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 48 M 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 264 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 762 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 221 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 27 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 18 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-29T04:06:29.891 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 32 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 746 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 127 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 162 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 321 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 298 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 99 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 85 k 2024-03-29T04:06:29.892 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 3.1 M 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 170 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-29T04:06:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 3.4 M 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 3.1 M 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-29T04:06:29.894 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 21 M 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 34 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 2.4 M 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 249 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 4.7 M 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 17 M 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2520.g69e2cd73.el9 ceph-noarch 16 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 25 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T04:06:29.895 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 163 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 500 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 5.1 M 2024-03-29T04:06:29.896 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T04:06:29.897 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 45 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-2520.g69e2cd73.el9 ceph 134 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T04:06:29.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T04:06:29.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-29T04:06:29.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T04:06:29.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-29T04:06:29.902 INFO:teuthology.orchestra.run.smithi156.stdout:Install 150 Packages 2024-03-29T04:06:29.903 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 24 Packages 2024-03-29T04:06:29.903 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:06:29.903 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 226 M 2024-03-29T04:06:29.903 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-29T04:06:30.412 INFO:teuthology.orchestra.run.smithi203.stdout:(1/174): ceph-19.0.0-2520.g69e2cd73.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-03-29T04:06:30.589 INFO:teuthology.orchestra.run.smithi156.stdout:(1/174): ceph-19.0.0-2520.g69e2cd73.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-03-29T04:06:30.863 INFO:teuthology.orchestra.run.smithi203.stdout:(2/174): ceph-base-19.0.0-2520.g69e2cd73.el9.x8 8.0 MB/s | 5.4 MB 00:00 2024-03-29T04:06:30.938 INFO:teuthology.orchestra.run.smithi203.stdout:(3/174): ceph-immutable-object-cache-19.0.0-252 1.9 MB/s | 145 kB 00:00 2024-03-29T04:06:31.213 INFO:teuthology.orchestra.run.smithi203.stdout:(4/174): ceph-mds-19.0.0-2520.g69e2cd73.el9.x86 8.6 MB/s | 2.4 MB 00:00 2024-03-29T04:06:31.447 INFO:teuthology.orchestra.run.smithi203.stdout:(5/174): ceph-fuse-19.0.0-2520.g69e2cd73.el9.x8 861 kB/s | 890 kB 00:01 2024-03-29T04:06:31.480 INFO:teuthology.orchestra.run.smithi203.stdout:(6/174): ceph-mgr-19.0.0-2520.g69e2cd73.el9.x86 5.6 MB/s | 1.5 MB 00:00 2024-03-29T04:06:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:(2/174): ceph-fuse-19.0.0-2520.g69e2cd73.el9.x8 962 kB/s | 890 kB 00:00 2024-03-29T04:06:31.690 INFO:teuthology.orchestra.run.smithi156.stdout:(3/174): ceph-immutable-object-cache-19.0.0-252 828 kB/s | 145 kB 00:00 2024-03-29T04:06:32.005 INFO:teuthology.orchestra.run.smithi203.stdout:(7/174): ceph-common-19.0.0-2520.g69e2cd73.el9. 11 MB/s | 21 MB 00:01 2024-03-29T04:06:32.281 INFO:teuthology.orchestra.run.smithi156.stdout:(4/174): ceph-base-19.0.0-2520.g69e2cd73.el9.x8 2.7 MB/s | 5.4 MB 00:01 2024-03-29T04:06:32.682 INFO:teuthology.orchestra.run.smithi156.stdout:(5/174): ceph-mgr-19.0.0-2520.g69e2cd73.el9.x86 3.7 MB/s | 1.5 MB 00:00 2024-03-29T04:06:32.749 INFO:teuthology.orchestra.run.smithi156.stdout:(6/174): ceph-mds-19.0.0-2520.g69e2cd73.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-03-29T04:06:32.955 INFO:teuthology.orchestra.run.smithi203.stdout:(8/174): ceph-radosgw-19.0.0-2520.g69e2cd73.el9 10 MB/s | 9.8 MB 00:00 2024-03-29T04:06:33.014 INFO:teuthology.orchestra.run.smithi203.stdout:(9/174): ceph-selinux-19.0.0-2520.g69e2cd73.el9 432 kB/s | 25 kB 00:00 2024-03-29T04:06:33.674 INFO:teuthology.orchestra.run.smithi156.stdout:(7/174): ceph-mon-19.0.0-2520.g69e2cd73.el9.x86 4.7 MB/s | 4.7 MB 00:00 2024-03-29T04:06:33.848 INFO:teuthology.orchestra.run.smithi203.stdout:(10/174): ceph-osd-19.0.0-2520.g69e2cd73.el9.x8 7.1 MB/s | 17 MB 00:02 2024-03-29T04:06:33.948 INFO:teuthology.orchestra.run.smithi203.stdout:(11/174): cephfs-mirror-19.0.0-2520.g69e2cd73.e 2.2 MB/s | 221 kB 00:00 2024-03-29T04:06:34.031 INFO:teuthology.orchestra.run.smithi203.stdout:(12/174): cephfs-shell-19.0.0-2520.g69e2cd73.el 325 kB/s | 27 kB 00:00 2024-03-29T04:06:34.115 INFO:teuthology.orchestra.run.smithi203.stdout:(13/174): libcephfs-devel-19.0.0-2520.g69e2cd73 379 kB/s | 32 kB 00:00 2024-03-29T04:06:34.291 INFO:teuthology.orchestra.run.smithi203.stdout:(14/174): libcephfs2-19.0.0-2520.g69e2cd73.el9. 4.2 MB/s | 746 kB 00:00 2024-03-29T04:06:34.391 INFO:teuthology.orchestra.run.smithi203.stdout:(15/174): libcephsqlite-19.0.0-2520.g69e2cd73.e 1.6 MB/s | 163 kB 00:00 2024-03-29T04:06:34.483 INFO:teuthology.orchestra.run.smithi203.stdout:(16/174): librados-devel-19.0.0-2520.g69e2cd73. 1.3 MB/s | 127 kB 00:00 2024-03-29T04:06:34.716 INFO:teuthology.orchestra.run.smithi203.stdout:(17/174): libradosstriper1-19.0.0-2520.g69e2cd7 2.1 MB/s | 500 kB 00:00 2024-03-29T04:06:35.808 INFO:teuthology.orchestra.run.smithi203.stdout:(18/174): librgw2-19.0.0-2520.g69e2cd73.el9.x86 4.7 MB/s | 5.1 MB 00:01 2024-03-29T04:06:35.891 INFO:teuthology.orchestra.run.smithi203.stdout:(19/174): python3-ceph-argparse-19.0.0-2520.g69 544 kB/s | 45 kB 00:00 2024-03-29T04:06:35.984 INFO:teuthology.orchestra.run.smithi203.stdout:(20/174): python3-ceph-common-19.0.0-2520.g69e2 1.4 MB/s | 134 kB 00:00 2024-03-29T04:06:36.067 INFO:teuthology.orchestra.run.smithi203.stdout:(21/174): python3-cephfs-19.0.0-2520.g69e2cd73. 1.9 MB/s | 162 kB 00:00 2024-03-29T04:06:36.151 INFO:teuthology.orchestra.run.smithi203.stdout:(22/174): python3-rados-19.0.0-2520.g69e2cd73.e 3.8 MB/s | 321 kB 00:00 2024-03-29T04:06:36.174 INFO:teuthology.orchestra.run.smithi156.stdout:(8/174): ceph-radosgw-19.0.0-2520.g69e2cd73.el9 3.9 MB/s | 9.8 MB 00:02 2024-03-29T04:06:36.210 INFO:teuthology.orchestra.run.smithi203.stdout:(23/174): python3-rbd-19.0.0-2520.g69e2cd73.el9 5.0 MB/s | 298 kB 00:00 2024-03-29T04:06:36.274 INFO:teuthology.orchestra.run.smithi156.stdout:(9/174): ceph-selinux-19.0.0-2520.g69e2cd73.el9 252 kB/s | 25 kB 00:00 2024-03-29T04:06:36.284 INFO:teuthology.orchestra.run.smithi203.stdout:(24/174): python3-rgw-19.0.0-2520.g69e2cd73.el9 1.3 MB/s | 99 kB 00:00 2024-03-29T04:06:36.360 INFO:teuthology.orchestra.run.smithi203.stdout:(25/174): rbd-fuse-19.0.0-2520.g69e2cd73.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-29T04:06:37.085 INFO:teuthology.orchestra.run.smithi203.stdout:(26/174): rbd-mirror-19.0.0-2520.g69e2cd73.el9. 4.3 MB/s | 3.1 MB 00:00 2024-03-29T04:06:37.152 INFO:teuthology.orchestra.run.smithi203.stdout:(27/174): ceph-mon-19.0.0-2520.g69e2cd73.el9.x8 843 kB/s | 4.7 MB 00:05 2024-03-29T04:06:37.177 INFO:teuthology.orchestra.run.smithi203.stdout:(28/174): rbd-nbd-19.0.0-2520.g69e2cd73.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-29T04:06:37.261 INFO:teuthology.orchestra.run.smithi203.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2520.g 313 kB/s | 34 kB 00:00 2024-03-29T04:06:37.286 INFO:teuthology.orchestra.run.smithi203.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2520.g69e2cd7 1.3 MB/s | 141 kB 00:00 2024-03-29T04:06:38.478 INFO:teuthology.orchestra.run.smithi203.stdout:(31/174): ceph-test-19.0.0-2520.g69e2cd73.el9.x 8.7 MB/s | 48 MB 00:05 2024-03-29T04:06:38.570 INFO:teuthology.orchestra.run.smithi203.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2520.g69 2.7 MB/s | 249 kB 00:00 2024-03-29T04:06:38.654 INFO:teuthology.orchestra.run.smithi203.stdout:(33/174): ceph-mgr-rook-19.0.0-2520.g69e2cd73.e 594 kB/s | 49 kB 00:00 2024-03-29T04:06:38.729 INFO:teuthology.orchestra.run.smithi203.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2520.g6 220 kB/s | 16 kB 00:00 2024-03-29T04:06:38.804 INFO:teuthology.orchestra.run.smithi203.stdout:(35/174): ceph-volume-19.0.0-2520.g69e2cd73.el9 3.5 MB/s | 264 kB 00:00 2024-03-29T04:06:38.891 INFO:teuthology.orchestra.run.smithi156.stdout:(10/174): ceph-common-19.0.0-2520.g69e2cd73.el9 2.4 MB/s | 21 MB 00:08 2024-03-29T04:06:38.983 INFO:teuthology.orchestra.run.smithi156.stdout:(11/174): cephfs-mirror-19.0.0-2520.g69e2cd73.e 2.4 MB/s | 221 kB 00:00 2024-03-29T04:06:39.033 INFO:teuthology.orchestra.run.smithi156.stdout:(12/174): cephfs-shell-19.0.0-2520.g69e2cd73.el 543 kB/s | 27 kB 00:00 2024-03-29T04:06:39.039 INFO:teuthology.orchestra.run.smithi203.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 4.2 MB/s | 7.4 MB 00:01 2024-03-29T04:06:39.080 INFO:teuthology.orchestra.run.smithi203.stdout:(37/174): cephadm-19.0.0-2520.g69e2cd73.el9.noa 2.7 MB/s | 762 kB 00:00 2024-03-29T04:06:39.083 INFO:teuthology.orchestra.run.smithi156.stdout:(13/174): libcephfs-devel-19.0.0-2520.g69e2cd73 636 kB/s | 32 kB 00:00 2024-03-29T04:06:39.097 INFO:teuthology.orchestra.run.smithi203.stdout:(38/174): cephfs-top-19.0.0-2520.g69e2cd73.el9. 317 kB/s | 18 kB 00:00 2024-03-29T04:06:39.247 INFO:teuthology.orchestra.run.smithi203.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-29T04:06:39.289 INFO:teuthology.orchestra.run.smithi203.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-29T04:06:39.314 INFO:teuthology.orchestra.run.smithi203.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-29T04:06:39.340 INFO:teuthology.orchestra.run.smithi203.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-29T04:06:39.350 INFO:teuthology.orchestra.run.smithi156.stdout:(14/174): libcephfs2-19.0.0-2520.g69e2cd73.el9. 2.7 MB/s | 746 kB 00:00 2024-03-29T04:06:39.356 INFO:teuthology.orchestra.run.smithi203.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 765 kB/s | 32 kB 00:00 2024-03-29T04:06:39.417 INFO:teuthology.orchestra.run.smithi156.stdout:(15/174): libcephsqlite-19.0.0-2520.g69e2cd73.e 2.4 MB/s | 163 kB 00:00 2024-03-29T04:06:39.440 INFO:teuthology.orchestra.run.smithi203.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-29T04:06:39.457 INFO:teuthology.orchestra.run.smithi203.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 172 kB/s | 17 kB 00:00 2024-03-29T04:06:39.482 INFO:teuthology.orchestra.run.smithi203.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 406 kB/s | 10 kB 00:00 2024-03-29T04:06:39.484 INFO:teuthology.orchestra.run.smithi156.stdout:(16/174): librados-devel-19.0.0-2520.g69e2cd73. 1.9 MB/s | 127 kB 00:00 2024-03-29T04:06:39.508 INFO:teuthology.orchestra.run.smithi203.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T04:06:39.525 INFO:teuthology.orchestra.run.smithi203.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T04:06:39.567 INFO:teuthology.orchestra.run.smithi203.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T04:06:39.592 INFO:teuthology.orchestra.run.smithi203.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-29T04:06:39.609 INFO:teuthology.orchestra.run.smithi203.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T04:06:39.634 INFO:teuthology.orchestra.run.smithi203.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T04:06:39.651 INFO:teuthology.orchestra.run.smithi156.stdout:(17/174): libradosstriper1-19.0.0-2520.g69e2cd7 2.9 MB/s | 500 kB 00:00 2024-03-29T04:06:39.659 INFO:teuthology.orchestra.run.smithi203.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-29T04:06:39.701 INFO:teuthology.orchestra.run.smithi203.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-29T04:06:39.718 INFO:teuthology.orchestra.run.smithi203.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 588 kB/s | 34 kB 00:00 2024-03-29T04:06:39.827 INFO:teuthology.orchestra.run.smithi203.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-29T04:06:39.869 INFO:teuthology.orchestra.run.smithi203.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-29T04:06:39.903 INFO:teuthology.orchestra.run.smithi203.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-03-29T04:06:39.920 INFO:teuthology.orchestra.run.smithi203.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 132 kB 00:00 2024-03-29T04:06:40.045 INFO:teuthology.orchestra.run.smithi203.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-29T04:06:40.078 INFO:teuthology.orchestra.run.smithi203.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T04:06:40.104 INFO:teuthology.orchestra.run.smithi203.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-29T04:06:40.221 INFO:teuthology.orchestra.run.smithi203.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-29T04:06:40.246 INFO:teuthology.orchestra.run.smithi203.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-29T04:06:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-29T04:06:40.296 INFO:teuthology.orchestra.run.smithi203.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T04:06:40.322 INFO:teuthology.orchestra.run.smithi203.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-29T04:06:40.422 INFO:teuthology.orchestra.run.smithi203.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-29T04:06:40.447 INFO:teuthology.orchestra.run.smithi203.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-29T04:06:40.473 INFO:teuthology.orchestra.run.smithi203.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-29T04:06:40.498 INFO:teuthology.orchestra.run.smithi203.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-29T04:06:40.524 INFO:teuthology.orchestra.run.smithi203.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-03-29T04:06:40.549 INFO:teuthology.orchestra.run.smithi203.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-29T04:06:40.574 INFO:teuthology.orchestra.run.smithi203.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-29T04:06:40.599 INFO:teuthology.orchestra.run.smithi203.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-29T04:06:40.616 INFO:teuthology.orchestra.run.smithi203.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-29T04:06:40.633 INFO:teuthology.orchestra.run.smithi203.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-29T04:06:40.700 INFO:teuthology.orchestra.run.smithi203.stdout:(78/174): ceph-mgr-dashboard-19.0.0-2520.g69e2c 1.0 MB/s | 3.5 MB 00:03 2024-03-29T04:06:40.784 INFO:teuthology.orchestra.run.smithi203.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 259 kB/s | 39 kB 00:00 2024-03-29T04:06:40.967 INFO:teuthology.orchestra.run.smithi156.stdout:(18/174): librgw2-19.0.0-2520.g69e2cd73.el9.x86 3.9 MB/s | 5.1 MB 00:01 2024-03-29T04:06:41.001 INFO:teuthology.orchestra.run.smithi203.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-29T04:06:41.026 INFO:teuthology.orchestra.run.smithi203.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 127 kB/s | 41 kB 00:00 2024-03-29T04:06:41.034 INFO:teuthology.orchestra.run.smithi156.stdout:(19/174): python3-ceph-argparse-19.0.0-2520.g69 680 kB/s | 45 kB 00:00 2024-03-29T04:06:41.051 INFO:teuthology.orchestra.run.smithi203.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 832 kB/s | 220 kB 00:00 2024-03-29T04:06:41.078 INFO:teuthology.orchestra.run.smithi203.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 466 kB/s | 35 kB 00:00 2024-03-29T04:06:41.102 INFO:teuthology.orchestra.run.smithi203.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T04:06:41.109 INFO:teuthology.orchestra.run.smithi156.stdout:(20/174): python3-ceph-common-19.0.0-2520.g69e2 1.7 MB/s | 134 kB 00:00 2024-03-29T04:06:41.127 INFO:teuthology.orchestra.run.smithi203.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-29T04:06:41.152 INFO:teuthology.orchestra.run.smithi203.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 752 kB/s | 56 kB 00:00 2024-03-29T04:06:41.177 INFO:teuthology.orchestra.run.smithi203.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T04:06:41.185 INFO:teuthology.orchestra.run.smithi156.stdout:(21/174): python3-cephfs-19.0.0-2520.g69e2cd73. 2.1 MB/s | 162 kB 00:00 2024-03-29T04:06:41.202 INFO:teuthology.orchestra.run.smithi203.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-29T04:06:41.228 INFO:teuthology.orchestra.run.smithi203.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-29T04:06:41.253 INFO:teuthology.orchestra.run.smithi203.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T04:06:41.286 INFO:teuthology.orchestra.run.smithi203.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-29T04:06:41.312 INFO:teuthology.orchestra.run.smithi203.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T04:06:41.337 INFO:teuthology.orchestra.run.smithi203.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-29T04:06:41.352 INFO:teuthology.orchestra.run.smithi156.stdout:(22/174): ceph-osd-19.0.0-2520.g69e2cd73.el9.x8 2.0 MB/s | 17 MB 00:08 2024-03-29T04:06:41.362 INFO:teuthology.orchestra.run.smithi203.stdout:(94/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-03-29T04:06:41.377 INFO:teuthology.orchestra.run.smithi156.stdout:(23/174): python3-rados-19.0.0-2520.g69e2cd73.e 1.6 MB/s | 321 kB 00:00 2024-03-29T04:06:41.452 INFO:teuthology.orchestra.run.smithi156.stdout:(24/174): python3-rgw-19.0.0-2520.g69e2cd73.el9 1.3 MB/s | 99 kB 00:00 2024-03-29T04:06:41.511 INFO:teuthology.orchestra.run.smithi156.stdout:(25/174): python3-rbd-19.0.0-2520.g69e2cd73.el9 1.8 MB/s | 298 kB 00:00 2024-03-29T04:06:41.528 INFO:teuthology.orchestra.run.smithi156.stdout:(26/174): rbd-fuse-19.0.0-2520.g69e2cd73.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-29T04:06:41.563 INFO:teuthology.orchestra.run.smithi203.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 749 kB/s | 187 kB 00:00 2024-03-29T04:06:41.603 INFO:teuthology.orchestra.run.smithi156.stdout:(27/174): rbd-nbd-19.0.0-2520.g69e2cd73.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-29T04:06:41.621 INFO:teuthology.orchestra.run.smithi203.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-29T04:06:41.647 INFO:teuthology.orchestra.run.smithi203.stdout:(97/174): libquadmath-11.4.1-3.el9.x86_64.rpm 688 kB/s | 195 kB 00:00 2024-03-29T04:06:41.670 INFO:teuthology.orchestra.run.smithi156.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2520.g 510 kB/s | 34 kB 00:00 2024-03-29T04:06:41.680 INFO:teuthology.orchestra.run.smithi203.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-29T04:06:41.712 INFO:teuthology.orchestra.run.smithi156.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2520.g69e2cd7 3.3 MB/s | 141 kB 00:00 2024-03-29T04:06:42.230 INFO:teuthology.orchestra.run.smithi203.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-03-29T04:06:42.255 INFO:teuthology.orchestra.run.smithi203.stdout:(100/174): elfutils-libelf-devel-0.190-2.el9.x8 34 kB/s | 21 kB 00:00 2024-03-29T04:06:42.529 INFO:teuthology.orchestra.run.smithi156.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2520.g69e2c 4.3 MB/s | 3.5 MB 00:00 2024-03-29T04:06:42.613 INFO:teuthology.orchestra.run.smithi156.stdout:(31/174): rbd-mirror-19.0.0-2520.g69e2cd73.el9. 2.8 MB/s | 3.1 MB 00:01 2024-03-29T04:06:42.780 INFO:teuthology.orchestra.run.smithi156.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2520.g69 1.5 MB/s | 249 kB 00:00 2024-03-29T04:06:42.880 INFO:teuthology.orchestra.run.smithi156.stdout:(33/174): ceph-mgr-rook-19.0.0-2520.g69e2cd73.e 493 kB/s | 49 kB 00:00 2024-03-29T04:06:42.963 INFO:teuthology.orchestra.run.smithi156.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2520.g6 197 kB/s | 16 kB 00:00 2024-03-29T04:06:43.114 INFO:teuthology.orchestra.run.smithi156.stdout:(35/174): ceph-volume-19.0.0-2520.g69e2cd73.el9 1.7 MB/s | 264 kB 00:00 2024-03-29T04:06:43.181 INFO:teuthology.orchestra.run.smithi203.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 258 kB/s | 245 kB 00:00 2024-03-29T04:06:43.206 INFO:teuthology.orchestra.run.smithi203.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 261 kB/s | 248 kB 00:00 2024-03-29T04:06:43.256 INFO:teuthology.orchestra.run.smithi203.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 451 kB/s | 22 kB 00:00 2024-03-29T04:06:43.407 INFO:teuthology.orchestra.run.smithi203.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-03-29T04:06:43.439 INFO:teuthology.orchestra.run.smithi156.stdout:(36/174): cephadm-19.0.0-2520.g69e2cd73.el9.noa 2.3 MB/s | 762 kB 00:00 2024-03-29T04:06:43.531 INFO:teuthology.orchestra.run.smithi156.stdout:(37/174): cephfs-top-19.0.0-2520.g69e2cd73.el9. 199 kB/s | 18 kB 00:00 2024-03-29T04:06:43.624 INFO:teuthology.orchestra.run.smithi203.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.8 MB/s | 548 kB 00:00 2024-03-29T04:06:43.674 INFO:teuthology.orchestra.run.smithi203.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 6.1 MB/s | 309 kB 00:00 2024-03-29T04:06:43.690 INFO:teuthology.orchestra.run.smithi156.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-29T04:06:43.707 INFO:teuthology.orchestra.run.smithi203.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 587 kB/s | 19 kB 00:00 2024-03-29T04:06:43.748 INFO:teuthology.orchestra.run.smithi156.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-29T04:06:43.798 INFO:teuthology.orchestra.run.smithi156.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-29T04:06:43.840 INFO:teuthology.orchestra.run.smithi156.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-29T04:06:43.849 INFO:teuthology.orchestra.run.smithi203.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-03-29T04:06:43.874 INFO:teuthology.orchestra.run.smithi156.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 947 kB/s | 32 kB 00:00 2024-03-29T04:06:43.883 INFO:teuthology.orchestra.run.smithi203.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 778 kB/s | 26 kB 00:00 2024-03-29T04:06:43.917 INFO:teuthology.orchestra.run.smithi203.stdout:(110/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 127 kB/s | 94 kB 00:00 2024-03-29T04:06:43.933 INFO:teuthology.orchestra.run.smithi203.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-03-29T04:06:43.966 INFO:teuthology.orchestra.run.smithi156.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-29T04:06:43.975 INFO:teuthology.orchestra.run.smithi203.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-03-29T04:06:44.000 INFO:teuthology.orchestra.run.smithi156.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 513 kB/s | 17 kB 00:00 2024-03-29T04:06:44.033 INFO:teuthology.orchestra.run.smithi156.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-29T04:06:44.051 INFO:teuthology.orchestra.run.smithi203.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-29T04:06:44.076 INFO:teuthology.orchestra.run.smithi203.stdout:(114/174): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-03-29T04:06:44.101 INFO:teuthology.orchestra.run.smithi203.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-29T04:06:44.127 INFO:teuthology.orchestra.run.smithi203.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 865 kB/s | 43 kB 00:00 2024-03-29T04:06:44.152 INFO:teuthology.orchestra.run.smithi203.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-03-29T04:06:44.177 INFO:teuthology.orchestra.run.smithi203.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-29T04:06:44.202 INFO:teuthology.orchestra.run.smithi203.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-29T04:06:44.244 INFO:teuthology.orchestra.run.smithi203.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-03-29T04:06:44.270 INFO:teuthology.orchestra.run.smithi203.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-29T04:06:44.287 INFO:teuthology.orchestra.run.smithi203.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 842 kB/s | 35 kB 00:00 2024-03-29T04:06:44.320 INFO:teuthology.orchestra.run.smithi203.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-03-29T04:06:44.370 INFO:teuthology.orchestra.run.smithi203.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 24 MB/s | 2.0 MB 00:00 2024-03-29T04:06:44.396 INFO:teuthology.orchestra.run.smithi203.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-29T04:06:44.413 INFO:teuthology.orchestra.run.smithi203.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-03-29T04:06:44.430 INFO:teuthology.orchestra.run.smithi203.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 528 kB/s | 18 kB 00:00 2024-03-29T04:06:44.447 INFO:teuthology.orchestra.run.smithi203.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 688 kB/s | 23 kB 00:00 2024-03-29T04:06:44.463 INFO:teuthology.orchestra.run.smithi203.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 578 kB/s | 19 kB 00:00 2024-03-29T04:06:44.475 INFO:teuthology.orchestra.run.smithi156.stdout:(46/174): ceph-test-19.0.0-2520.g69e2cd73.el9.x 5.8 MB/s | 48 MB 00:08 2024-03-29T04:06:44.480 INFO:teuthology.orchestra.run.smithi203.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 588 kB/s | 20 kB 00:00 2024-03-29T04:06:44.501 INFO:teuthology.orchestra.run.smithi156.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 341 kB/s | 159 kB 00:00 2024-03-29T04:06:44.523 INFO:teuthology.orchestra.run.smithi203.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-29T04:06:44.548 INFO:teuthology.orchestra.run.smithi203.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 689 kB/s | 46 kB 00:00 2024-03-29T04:06:44.573 INFO:teuthology.orchestra.run.smithi203.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-29T04:06:44.598 INFO:teuthology.orchestra.run.smithi203.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-29T04:06:44.601 INFO:teuthology.orchestra.run.smithi156.stdout:(48/174): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-03-29T04:06:44.624 INFO:teuthology.orchestra.run.smithi203.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T04:06:44.643 INFO:teuthology.orchestra.run.smithi156.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 265 kB/s | 44 kB 00:00 2024-03-29T04:06:44.650 INFO:teuthology.orchestra.run.smithi203.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-03-29T04:06:44.674 INFO:teuthology.orchestra.run.smithi203.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-29T04:06:44.676 INFO:teuthology.orchestra.run.smithi156.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-29T04:06:44.700 INFO:teuthology.orchestra.run.smithi203.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-03-29T04:06:44.701 INFO:teuthology.orchestra.run.smithi156.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 578 kB 00:00 2024-03-29T04:06:44.725 INFO:teuthology.orchestra.run.smithi203.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-03-29T04:06:44.726 INFO:teuthology.orchestra.run.smithi156.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 46 kB 00:00 2024-03-29T04:06:44.750 INFO:teuthology.orchestra.run.smithi203.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-29T04:06:44.752 INFO:teuthology.orchestra.run.smithi156.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T04:06:44.775 INFO:teuthology.orchestra.run.smithi203.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-29T04:06:44.777 INFO:teuthology.orchestra.run.smithi156.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-29T04:06:44.801 INFO:teuthology.orchestra.run.smithi203.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-03-29T04:06:44.802 INFO:teuthology.orchestra.run.smithi156.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-29T04:06:44.826 INFO:teuthology.orchestra.run.smithi203.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-29T04:06:44.828 INFO:teuthology.orchestra.run.smithi156.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-29T04:06:44.851 INFO:teuthology.orchestra.run.smithi203.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T04:06:44.876 INFO:teuthology.orchestra.run.smithi203.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-29T04:06:44.910 INFO:teuthology.orchestra.run.smithi203.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-03-29T04:06:44.928 INFO:teuthology.orchestra.run.smithi156.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-29T04:06:44.935 INFO:teuthology.orchestra.run.smithi203.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 340 kB/s | 20 kB 00:00 2024-03-29T04:06:44.961 INFO:teuthology.orchestra.run.smithi203.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-29T04:06:44.970 INFO:teuthology.orchestra.run.smithi156.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 9.3 MB/s | 1.5 MB 00:00 2024-03-29T04:06:45.003 INFO:teuthology.orchestra.run.smithi203.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-03-29T04:06:45.003 INFO:teuthology.orchestra.run.smithi156.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 4.6 MB/s | 817 kB 00:00 2024-03-29T04:06:45.037 INFO:teuthology.orchestra.run.smithi156.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 132 kB 00:00 2024-03-29T04:06:45.079 INFO:teuthology.orchestra.run.smithi156.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-29T04:06:45.205 INFO:teuthology.orchestra.run.smithi156.stdout:(62/174): postgresql-server-13.14-1.el9.x86_64. 29 MB/s | 5.7 MB 00:00 2024-03-29T04:06:45.230 INFO:teuthology.orchestra.run.smithi156.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 408 kB/s | 78 kB 00:00 2024-03-29T04:06:45.328 INFO:teuthology.orchestra.run.smithi203.stdout:(150/174): librbd1-19.0.0-2520.g69e2cd73.el9.x8 9.7 MB/s | 3.1 MB 00:00 2024-03-29T04:06:45.330 INFO:teuthology.orchestra.run.smithi156.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-29T04:06:45.347 INFO:teuthology.orchestra.run.smithi156.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-29T04:06:45.363 INFO:teuthology.orchestra.run.smithi156.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-29T04:06:45.389 INFO:teuthology.orchestra.run.smithi156.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-29T04:06:45.420 INFO:teuthology.orchestra.run.smithi203.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 25 MB/s | 2.3 MB 00:00 2024-03-29T04:06:45.422 INFO:teuthology.orchestra.run.smithi156.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 541 kB/s | 32 kB 00:00 2024-03-29T04:06:45.448 INFO:teuthology.orchestra.run.smithi156.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 217 kB 00:00 2024-03-29T04:06:45.512 INFO:teuthology.orchestra.run.smithi203.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-03-29T04:06:45.545 INFO:teuthology.orchestra.run.smithi203.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-29T04:06:45.556 INFO:teuthology.orchestra.run.smithi156.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-29T04:06:45.581 INFO:teuthology.orchestra.run.smithi156.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-29T04:06:45.588 INFO:teuthology.orchestra.run.smithi203.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-29T04:06:45.607 INFO:teuthology.orchestra.run.smithi156.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 439 kB/s | 69 kB 00:00 2024-03-29T04:06:45.624 INFO:teuthology.orchestra.run.smithi156.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 619 kB/s | 41 kB 00:00 2024-03-29T04:06:45.629 INFO:teuthology.orchestra.run.smithi203.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-29T04:06:45.649 INFO:teuthology.orchestra.run.smithi156.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-29T04:06:45.666 INFO:teuthology.orchestra.run.smithi156.stdout:(75/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-29T04:06:45.691 INFO:teuthology.orchestra.run.smithi156.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-29T04:06:45.708 INFO:teuthology.orchestra.run.smithi156.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 817 kB/s | 47 kB 00:00 2024-03-29T04:06:45.725 INFO:teuthology.orchestra.run.smithi156.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-29T04:06:45.763 INFO:teuthology.orchestra.run.smithi203.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 535 kB/s | 71 kB 00:00 2024-03-29T04:06:45.830 INFO:teuthology.orchestra.run.smithi203.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 547 kB/s | 36 kB 00:00 2024-03-29T04:06:45.950 INFO:teuthology.orchestra.run.smithi156.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 160 kB/s | 39 kB 00:00 2024-03-29T04:06:45.967 INFO:teuthology.orchestra.run.smithi156.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 171 kB/s | 41 kB 00:00 2024-03-29T04:06:46.014 INFO:teuthology.orchestra.run.smithi203.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.0 MB/s | 192 kB 00:00 2024-03-29T04:06:46.139 INFO:teuthology.orchestra.run.smithi203.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-29T04:06:46.142 INFO:teuthology.orchestra.run.smithi156.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-29T04:06:46.167 INFO:teuthology.orchestra.run.smithi156.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-03-29T04:06:46.193 INFO:teuthology.orchestra.run.smithi156.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 156 kB/s | 35 kB 00:00 2024-03-29T04:06:46.214 INFO:teuthology.orchestra.run.smithi203.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-29T04:06:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 620 kB/s | 46 kB 00:00 2024-03-29T04:06:46.243 INFO:teuthology.orchestra.run.smithi156.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T04:06:46.268 INFO:teuthology.orchestra.run.smithi156.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 752 kB/s | 56 kB 00:00 2024-03-29T04:06:46.294 INFO:teuthology.orchestra.run.smithi156.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 837 kB/s | 63 kB 00:00 2024-03-29T04:06:46.306 INFO:teuthology.orchestra.run.smithi203.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-29T04:06:46.319 INFO:teuthology.orchestra.run.smithi156.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-29T04:06:46.344 INFO:teuthology.orchestra.run.smithi156.stdout:(89/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-29T04:06:46.369 INFO:teuthology.orchestra.run.smithi156.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-29T04:06:46.403 INFO:teuthology.orchestra.run.smithi156.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-29T04:06:46.406 INFO:teuthology.orchestra.run.smithi203.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-29T04:06:46.428 INFO:teuthology.orchestra.run.smithi156.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-29T04:06:46.453 INFO:teuthology.orchestra.run.smithi156.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-29T04:06:46.479 INFO:teuthology.orchestra.run.smithi156.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-29T04:06:46.498 INFO:teuthology.orchestra.run.smithi203.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-29T04:06:46.554 INFO:teuthology.orchestra.run.smithi156.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-03-29T04:06:46.579 INFO:teuthology.orchestra.run.smithi156.stdout:(96/174): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T04:06:46.610 INFO:teuthology.orchestra.run.smithi203.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 571 kB/s | 64 kB 00:00 2024-03-29T04:06:46.630 INFO:teuthology.orchestra.run.smithi156.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T04:06:46.663 INFO:teuthology.orchestra.run.smithi156.stdout:(98/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-29T04:06:46.682 INFO:teuthology.orchestra.run.smithi203.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 402 kB/s | 29 kB 00:00 2024-03-29T04:06:46.688 INFO:teuthology.orchestra.run.smithi156.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-29T04:06:46.722 INFO:teuthology.orchestra.run.smithi156.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 276 kB/s | 25 kB 00:00 2024-03-29T04:06:46.741 INFO:teuthology.orchestra.run.smithi203.stdout:(166/174): librados2-19.0.0-2520.g69e2cd73.el9. 1.9 MB/s | 3.4 MB 00:01 2024-03-29T04:06:46.789 INFO:teuthology.orchestra.run.smithi156.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.4 MB/s | 245 kB 00:00 2024-03-29T04:06:46.823 INFO:teuthology.orchestra.run.smithi156.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.4 MB/s | 248 kB 00:00 2024-03-29T04:06:46.848 INFO:teuthology.orchestra.run.smithi156.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-29T04:06:46.973 INFO:teuthology.orchestra.run.smithi156.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-03-29T04:06:46.990 INFO:teuthology.orchestra.run.smithi156.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-03-29T04:06:47.049 INFO:teuthology.orchestra.run.smithi203.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 3.3 MB/s | 1.2 MB 00:00 2024-03-29T04:06:47.150 INFO:teuthology.orchestra.run.smithi203.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.3 MB/s | 2.2 MB 00:00 2024-03-29T04:06:47.215 INFO:teuthology.orchestra.run.smithi156.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.2 MB/s | 548 kB 00:00 2024-03-29T04:06:47.241 INFO:teuthology.orchestra.run.smithi156.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-29T04:06:47.250 INFO:teuthology.orchestra.run.smithi203.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.6 MB/s | 465 kB 00:00 2024-03-29T04:06:47.257 INFO:teuthology.orchestra.run.smithi156.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-29T04:06:47.291 INFO:teuthology.orchestra.run.smithi156.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-03-29T04:06:47.309 INFO:teuthology.orchestra.run.smithi203.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-03-29T04:06:47.366 INFO:teuthology.orchestra.run.smithi156.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-03-29T04:06:47.392 INFO:teuthology.orchestra.run.smithi156.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-29T04:06:47.451 INFO:teuthology.orchestra.run.smithi203.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-29T04:06:47.458 INFO:teuthology.orchestra.run.smithi156.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T04:06:47.510 INFO:teuthology.orchestra.run.smithi156.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-29T04:06:47.534 INFO:teuthology.orchestra.run.smithi203.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.7 MB/s | 2.3 MB 00:00 2024-03-29T04:06:47.834 INFO:teuthology.orchestra.run.smithi156.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-29T04:06:47.859 INFO:teuthology.orchestra.run.smithi156.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 1.5 MB/s | 548 kB 00:00 2024-03-29T04:06:47.918 INFO:teuthology.orchestra.run.smithi156.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 869 kB/s | 43 kB 00:00 2024-03-29T04:06:47.926 INFO:teuthology.orchestra.run.smithi156.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 485 kB/s | 32 kB 00:00 2024-03-29T04:06:47.952 INFO:teuthology.orchestra.run.smithi156.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 226 kB/s | 15 kB 00:00 2024-03-29T04:06:48.043 INFO:teuthology.orchestra.run.smithi156.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-29T04:06:48.119 INFO:teuthology.orchestra.run.smithi156.stdout:(120/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-03-29T04:06:48.126 INFO:teuthology.orchestra.run.smithi203.stdout:(173/174): openssl-devel-3.0.7-27.el9.x86_64.rp 656 kB/s | 4.1 MB 00:06 2024-03-29T04:06:48.151 INFO:teuthology.orchestra.run.smithi203.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 106 kB/s | 74 kB 00:00 2024-03-29T04:06:48.152 INFO:teuthology.orchestra.run.smithi156.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-03-29T04:06:48.159 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T04:06:48.159 INFO:teuthology.orchestra.run.smithi203.stdout:Total 12 MB/s | 226 MB 00:18 2024-03-29T04:06:48.186 INFO:teuthology.orchestra.run.smithi156.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 527 kB/s | 35 kB 00:00 2024-03-29T04:06:48.211 INFO:teuthology.orchestra.run.smithi156.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-03-29T04:06:48.236 INFO:teuthology.orchestra.run.smithi156.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-29T04:06:48.287 INFO:teuthology.orchestra.run.smithi156.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 216 kB/s | 11 kB 00:00 2024-03-29T04:06:48.338 INFO:teuthology.orchestra.run.smithi156.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-29T04:06:48.379 INFO:teuthology.orchestra.run.smithi156.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-03-29T04:06:48.405 INFO:teuthology.orchestra.run.smithi156.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-03-29T04:06:48.446 INFO:teuthology.orchestra.run.smithi156.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 293 kB/s | 19 kB 00:00 2024-03-29T04:06:48.472 INFO:teuthology.orchestra.run.smithi156.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 297 kB/s | 20 kB 00:00 2024-03-29T04:06:48.539 INFO:teuthology.orchestra.run.smithi156.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 701 kB/s | 46 kB 00:00 2024-03-29T04:06:48.544 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T04:06:48.597 INFO:teuthology.orchestra.run.smithi156.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.0 MB/s | 2.0 MB 00:00 2024-03-29T04:06:48.622 INFO:teuthology.orchestra.run.smithi156.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 949 kB/s | 79 kB 00:00 2024-03-29T04:06:48.673 INFO:teuthology.orchestra.run.smithi156.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-03-29T04:06:48.698 INFO:teuthology.orchestra.run.smithi156.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 578 kB/s | 58 kB 00:00 2024-03-29T04:06:48.731 INFO:teuthology.orchestra.run.smithi156.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-29T04:06:48.822 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T04:06:48.822 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T04:06:48.840 INFO:teuthology.orchestra.run.smithi156.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2024-03-29T04:06:48.957 INFO:teuthology.orchestra.run.smithi156.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 348 kB/s | 90 kB 00:00 2024-03-29T04:06:48.982 INFO:teuthology.orchestra.run.smithi156.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 113 kB/s | 28 kB 00:00 2024-03-29T04:06:49.007 INFO:teuthology.orchestra.run.smithi156.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 184 kB/s | 31 kB 00:00 2024-03-29T04:06:49.041 INFO:teuthology.orchestra.run.smithi156.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-29T04:06:49.066 INFO:teuthology.orchestra.run.smithi156.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-03-29T04:06:49.092 INFO:teuthology.orchestra.run.smithi156.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-03-29T04:06:49.117 INFO:teuthology.orchestra.run.smithi156.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-03-29T04:06:49.142 INFO:teuthology.orchestra.run.smithi156.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T04:06:49.167 INFO:teuthology.orchestra.run.smithi156.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-29T04:06:49.209 INFO:teuthology.orchestra.run.smithi156.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-03-29T04:06:49.234 INFO:teuthology.orchestra.run.smithi156.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-03-29T04:06:49.260 INFO:teuthology.orchestra.run.smithi156.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-29T04:06:49.418 INFO:teuthology.orchestra.run.smithi156.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-29T04:06:49.510 INFO:teuthology.orchestra.run.smithi156.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 25 MB/s | 2.3 MB 00:00 2024-03-29T04:06:49.585 INFO:teuthology.orchestra.run.smithi156.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-29T04:06:49.619 INFO:teuthology.orchestra.run.smithi156.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-29T04:06:49.677 INFO:teuthology.orchestra.run.smithi156.stdout:(154/174): librbd1-19.0.0-2520.g69e2cd73.el9.x8 7.5 MB/s | 3.1 MB 00:00 2024-03-29T04:06:49.703 INFO:teuthology.orchestra.run.smithi156.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 247 kB 00:00 2024-03-29T04:06:49.736 INFO:teuthology.orchestra.run.smithi156.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-29T04:06:49.787 INFO:teuthology.orchestra.run.smithi156.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 729 kB/s | 36 kB 00:00 2024-03-29T04:06:49.821 INFO:teuthology.orchestra.run.smithi156.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 611 kB/s | 71 kB 00:00 2024-03-29T04:06:49.871 INFO:teuthology.orchestra.run.smithi156.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-29T04:06:49.929 INFO:teuthology.orchestra.run.smithi156.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-29T04:06:49.946 INFO:teuthology.orchestra.run.smithi156.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 307 kB/s | 23 kB 00:00 2024-03-29T04:06:50.005 INFO:teuthology.orchestra.run.smithi156.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-29T04:06:50.030 INFO:teuthology.orchestra.run.smithi156.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-29T04:06:50.089 INFO:teuthology.orchestra.run.smithi156.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-29T04:06:50.114 INFO:teuthology.orchestra.run.smithi156.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 765 kB/s | 64 kB 00:00 2024-03-29T04:06:50.156 INFO:teuthology.orchestra.run.smithi156.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 431 kB/s | 29 kB 00:00 2024-03-29T04:06:50.215 INFO:teuthology.orchestra.run.smithi156.stdout:(167/174): librados2-19.0.0-2520.g69e2cd73.el9. 3.5 MB/s | 3.4 MB 00:00 2024-03-29T04:06:50.348 INFO:teuthology.orchestra.run.smithi156.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 1.2 MB 00:00 2024-03-29T04:06:50.457 INFO:teuthology.orchestra.run.smithi156.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-29T04:06:50.499 INFO:teuthology.orchestra.run.smithi156.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-29T04:06:50.540 INFO:teuthology.orchestra.run.smithi156.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.0 MB/s | 2.3 MB 00:00 2024-03-29T04:06:50.566 INFO:teuthology.orchestra.run.smithi156.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 286 kB/s | 31 kB 00:00 2024-03-29T04:06:50.616 INFO:teuthology.orchestra.run.smithi156.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 992 kB/s | 74 kB 00:00 2024-03-29T04:06:50.691 INFO:teuthology.orchestra.run.smithi156.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-03-29T04:06:50.697 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-29T04:06:50.697 INFO:teuthology.orchestra.run.smithi156.stdout:Total 11 MB/s | 226 MB 00:20 2024-03-29T04:06:50.844 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T04:06:50.845 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T04:06:51.070 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-29T04:06:51.346 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-29T04:06:51.347 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-29T04:06:53.370 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-29T04:06:53.371 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-29T04:06:57.754 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T04:06:57.864 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-29T04:06:58.100 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-29T04:06:58.245 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-29T04:06:58.257 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-29T04:06:58.306 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-29T04:06:58.361 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-29T04:06:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-29T04:06:58.424 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-29T04:06:58.482 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-29T04:06:58.812 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-29T04:06:58.871 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 10/198 2024-03-29T04:06:58.993 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 10/198 2024-03-29T04:06:59.022 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 11/198 2024-03-29T04:06:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 11/198 2024-03-29T04:06:59.144 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-2520.g69e2cd73.el9.x86_64 12/198 2024-03-29T04:06:59.185 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-29T04:06:59.362 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-29T04:06:59.416 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-29T04:06:59.444 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 16/198 2024-03-29T04:06:59.548 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 16/198 2024-03-29T04:06:59.570 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86 17/198 2024-03-29T04:06:59.627 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86 17/198 2024-03-29T04:06:59.666 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-29T04:06:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-29T04:06:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-29T04:06:59.904 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-29T04:06:59.937 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-29T04:06:59.977 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-29T04:07:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-29T04:07:00.310 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-29T04:07:00.362 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-29T04:07:00.463 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-29T04:07:00.513 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-29T04:07:00.514 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-29T04:07:00.612 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-29T04:07:00.655 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-29T04:07:00.693 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-29T04:07:00.703 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-29T04:07:00.735 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-29T04:07:00.743 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-29T04:07:00.797 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-29T04:07:00.830 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-29T04:07:00.844 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-29T04:07:00.866 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-29T04:07:00.912 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-29T04:07:00.938 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-29T04:07:00.986 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-29T04:07:01.013 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-29T04:07:01.049 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-29T04:07:01.121 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-29T04:07:01.176 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-29T04:07:01.214 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-29T04:07:01.247 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-29T04:07:01.286 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-29T04:07:01.309 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 10/198 2024-03-29T04:07:01.322 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-29T04:07:01.365 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-29T04:07:01.408 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-29T04:07:01.434 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 10/198 2024-03-29T04:07:01.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-29T04:07:01.460 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 11/198 2024-03-29T04:07:01.506 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-29T04:07:01.540 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-2520.g69e2cd73.el9. 39/198 2024-03-29T04:07:01.552 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 11/198 2024-03-29T04:07:01.581 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-2520.g69e2cd73.el9.x86_64 12/198 2024-03-29T04:07:01.584 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-2520.g69e2cd73.el 40/198 2024-03-29T04:07:01.612 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-2520.g69e2cd73.el9.x86_6 41/198 2024-03-29T04:07:01.620 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-29T04:07:01.681 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-29T04:07:01.727 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-29T04:07:01.800 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-29T04:07:01.816 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-29T04:07:01.879 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-29T04:07:01.907 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 16/198 2024-03-29T04:07:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-29T04:07:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-29T04:07:02.002 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 16/198 2024-03-29T04:07:02.024 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86 17/198 2024-03-29T04:07:02.073 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86 17/198 2024-03-29T04:07:02.112 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-29T04:07:02.134 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-29T04:07:02.150 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-29T04:07:02.361 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-29T04:07:02.391 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-29T04:07:02.422 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-29T04:07:02.514 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-29T04:07:02.592 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-29T04:07:02.623 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-29T04:07:02.671 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-29T04:07:02.721 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-29T04:07:02.731 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-29T04:07:02.750 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-29T04:07:02.782 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-29T04:07:02.801 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-29T04:07:02.819 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-29T04:07:02.856 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-2520.g69e2cd73.el9.x86_6 53/198 2024-03-29T04:07:02.891 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-29T04:07:02.902 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-29T04:07:02.944 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-29T04:07:03.156 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-29T04:07:03.250 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-29T04:07:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-29T04:07:03.439 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-29T04:07:03.491 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-29T04:07:03.550 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-29T04:07:03.633 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-29T04:07:03.646 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-29T04:07:03.671 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-29T04:07:03.691 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-29T04:07:03.725 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-29T04:07:03.727 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-29T04:07:03.787 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-29T04:07:03.797 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-29T04:07:03.830 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-29T04:07:03.860 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-29T04:07:03.891 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-29T04:07:03.915 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-29T04:07:03.938 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-29T04:07:03.953 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-29T04:07:03.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-29T04:07:04.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-29T04:07:04.018 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-29T04:07:04.059 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-29T04:07:04.060 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-29T04:07:04.104 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-29T04:07:04.137 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-29T04:07:04.164 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-29T04:07:04.194 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-29T04:07:04.226 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-29T04:07:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-29T04:07:04.261 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-2520.g69e2cd73.el9. 39/198 2024-03-29T04:07:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-29T04:07:04.304 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-2520.g69e2cd73.el 40/198 2024-03-29T04:07:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-29T04:07:04.333 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-2520.g69e2cd73.el9.x86_6 41/198 2024-03-29T04:07:04.402 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-29T04:07:04.406 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-29T04:07:04.448 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-29T04:07:04.454 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-29T04:07:04.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-29T04:07:04.520 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-29T04:07:04.521 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-29T04:07:04.544 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-29T04:07:04.622 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-29T04:07:04.658 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-29T04:07:04.710 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-29T04:07:04.745 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-29T04:07:04.845 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-29T04:07:04.887 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-29T04:07:04.935 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-29T04:07:05.001 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-29T04:07:05.183 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-29T04:07:05.225 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-29T04:07:05.240 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-29T04:07:05.322 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-29T04:07:05.353 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-29T04:07:05.398 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-29T04:07:05.418 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-29T04:07:05.431 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-29T04:07:05.461 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-29T04:07:05.467 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-29T04:07:05.477 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-29T04:07:05.490 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-29T04:07:05.517 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-29T04:07:05.541 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-29T04:07:05.565 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-29T04:07:05.580 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-29T04:07:05.603 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-2520.g69e2cd73.el9.x86_6 53/198 2024-03-29T04:07:05.611 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-29T04:07:05.642 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-29T04:07:05.642 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-29T04:07:05.718 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-29T04:07:05.752 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-29T04:07:05.786 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-29T04:07:05.836 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-29T04:07:05.870 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-29T04:07:05.922 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-29T04:07:05.958 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-29T04:07:06.049 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-29T04:07:06.093 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-29T04:07:06.108 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-29T04:07:06.125 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-29T04:07:06.172 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-29T04:07:06.219 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-29T04:07:06.277 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-29T04:07:06.314 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-29T04:07:06.335 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-29T04:07:06.459 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-29T04:07:06.499 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-29T04:07:06.554 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-29T04:07:06.636 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-29T04:07:06.670 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-29T04:07:06.703 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-29T04:07:06.751 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-29T04:07:06.786 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-29T04:07:06.829 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-29T04:07:06.871 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-29T04:07:07.006 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-29T04:07:07.052 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-29T04:07:07.060 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-29T04:07:07.099 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-29T04:07:07.107 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-29T04:07:07.140 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-29T04:07:07.141 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-29T04:07:07.189 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-29T04:07:07.235 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-29T04:07:07.266 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-29T04:07:07.276 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-29T04:07:07.306 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-29T04:07:07.317 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-29T04:07:07.332 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-29T04:07:07.373 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-29T04:07:07.451 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-29T04:07:07.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-29T04:07:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-29T04:07:07.583 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-29T04:07:07.618 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-29T04:07:07.699 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-29T04:07:07.740 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-29T04:07:07.789 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-29T04:07:07.844 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-29T04:07:08.028 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-29T04:07:08.078 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-29T04:07:08.206 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-29T04:07:08.251 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-29T04:07:08.260 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-29T04:07:08.283 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-29T04:07:08.314 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-29T04:07:08.351 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-29T04:07:08.371 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-29T04:07:08.385 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-29T04:07:08.424 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-29T04:07:08.464 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-29T04:07:08.494 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-29T04:07:08.571 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-29T04:07:08.604 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-29T04:07:08.638 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-29T04:07:08.679 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-29T04:07:08.714 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-29T04:07:08.803 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-29T04:07:08.893 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-29T04:07:08.936 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-29T04:07:08.968 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-29T04:07:09.015 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-29T04:07:09.112 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-29T04:07:09.161 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-29T04:07:09.590 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-29T04:07:09.677 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-29T04:07:09.708 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-29T04:07:09.739 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-29T04:07:09.767 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-29T04:07:09.899 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-29T04:07:09.950 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-29T04:07:09.984 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-29T04:07:10.029 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-29T04:07:10.079 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-29T04:07:10.105 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 124/198 2024-03-29T04:07:10.116 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-29T04:07:10.157 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-29T04:07:10.179 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 124/198 2024-03-29T04:07:10.210 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 125/198 2024-03-29T04:07:10.246 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-29T04:07:10.335 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-29T04:07:10.371 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-29T04:07:10.467 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-29T04:07:10.707 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-29T04:07:10.947 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 128/198 2024-03-29T04:07:11.105 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-29T04:07:11.113 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 128/198 2024-03-29T04:07:11.136 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 129/198 2024-03-29T04:07:11.219 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-29T04:07:12.461 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-29T04:07:12.539 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-29T04:07:12.570 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-29T04:07:12.602 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-29T04:07:12.629 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-29T04:07:12.953 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-29T04:07:12.984 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 124/198 2024-03-29T04:07:13.075 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 124/198 2024-03-29T04:07:13.105 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 125/198 2024-03-29T04:07:13.141 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-29T04:07:13.449 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 130/198 2024-03-29T04:07:13.493 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 130/198 2024-03-29T04:07:13.611 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-29T04:07:13.637 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 128/198 2024-03-29T04:07:13.708 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 128/198 2024-03-29T04:07:13.748 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 129/198 2024-03-29T04:07:14.040 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 130/198 2024-03-29T04:07:14.071 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 131/198 2024-03-29T04:07:14.162 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 131/198 2024-03-29T04:07:14.162 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T04:07:14.162 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T04:07:14.163 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 132/198 2024-03-29T04:07:16.026 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 130/198 2024-03-29T04:07:16.061 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 130/198 2024-03-29T04:07:16.613 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 130/198 2024-03-29T04:07:16.648 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 131/198 2024-03-29T04:07:16.742 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 131/198 2024-03-29T04:07:16.742 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T04:07:16.742 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T04:07:16.742 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:16.790 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 132/198 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 132/198 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-03-29T04:07:26.239 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:26.409 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 133/198 2024-03-29T04:07:26.425 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 133/198 2024-03-29T04:07:26.425 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:26.425 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T04:07:26.425 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:27.920 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 134/198 2024-03-29T04:07:27.946 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 134/198 2024-03-29T04:07:27.946 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:27.946 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T04:07:27.946 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T04:07:27.946 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T04:07:27.946 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:28.171 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 135/198 2024-03-29T04:07:28.194 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 135/198 2024-03-29T04:07:28.194 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:28.194 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T04:07:28.194 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T04:07:28.194 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T04:07:28.195 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:28.630 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 136/198 2024-03-29T04:07:28.653 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 136/198 2024-03-29T04:07:28.653 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:28.653 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T04:07:28.653 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T04:07:28.653 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T04:07:28.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:28.749 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-29T04:07:29.127 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 138/198 2024-03-29T04:07:29.156 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 138/198 2024-03-29T04:07:29.188 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2520.g69e2cd73.e 139/198 2024-03-29T04:07:29.250 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2520.g69e2cd73. 140/198 2024-03-29T04:07:29.280 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noa 141/198 2024-03-29T04:07:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 132/198 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-03-29T04:07:29.300 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:29.464 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 133/198 2024-03-29T04:07:29.480 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 133/198 2024-03-29T04:07:29.480 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:29.480 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T04:07:29.480 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:30.319 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noa 141/198 2024-03-29T04:07:30.356 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.n 142/198 2024-03-29T04:07:30.960 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 134/198 2024-03-29T04:07:30.986 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 134/198 2024-03-29T04:07:30.986 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:30.986 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T04:07:30.986 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T04:07:30.986 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T04:07:30.986 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:31.210 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 135/198 2024-03-29T04:07:31.233 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 135/198 2024-03-29T04:07:31.233 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:31.233 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T04:07:31.233 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T04:07:31.233 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T04:07:31.234 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:31.321 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.n 142/198 2024-03-29T04:07:31.349 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e 143/198 2024-03-29T04:07:31.449 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e 143/198 2024-03-29T04:07:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2520.g69e2cd73.el 144/198 2024-03-29T04:07:31.651 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 145/198 2024-03-29T04:07:31.669 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 136/198 2024-03-29T04:07:31.674 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 145/198 2024-03-29T04:07:31.674 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:31.674 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T04:07:31.675 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T04:07:31.675 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T04:07:31.675 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:31.692 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 136/198 2024-03-29T04:07:31.692 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:31.692 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T04:07:31.692 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T04:07:31.693 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T04:07:31.693 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:31.719 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 146/198 2024-03-29T04:07:31.730 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 146/198 2024-03-29T04:07:31.784 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-2520.g69e2cd73.el9.x86_64 147/198 2024-03-29T04:07:31.796 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-29T04:07:32.249 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 138/198 2024-03-29T04:07:32.279 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 138/198 2024-03-29T04:07:32.315 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2520.g69e2cd73.e 139/198 2024-03-29T04:07:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2520.g69e2cd73. 140/198 2024-03-29T04:07:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noa 141/198 2024-03-29T04:07:32.657 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-29T04:07:32.688 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 149/198 2024-03-29T04:07:32.711 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 149/198 2024-03-29T04:07:32.711 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:32.711 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T04:07:32.711 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T04:07:32.711 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T04:07:32.711 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:32.755 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2520.g69e2c 150/198 2024-03-29T04:07:32.777 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2520.g69e2c 150/198 2024-03-29T04:07:32.778 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:32.778 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T04:07:32.778 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:32.831 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 151/198 2024-03-29T04:07:32.853 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 151/198 2024-03-29T04:07:32.853 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:32.853 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-29T04:07:32.853 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-29T04:07:32.854 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-29T04:07:32.854 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:33.165 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 152/198 2024-03-29T04:07:33.187 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 152/198 2024-03-29T04:07:33.187 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:33.187 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T04:07:33.188 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T04:07:33.188 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T04:07:33.188 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:33.453 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noa 141/198 2024-03-29T04:07:33.512 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.n 142/198 2024-03-29T04:07:34.474 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.n 142/198 2024-03-29T04:07:34.504 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e 143/198 2024-03-29T04:07:34.604 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e 143/198 2024-03-29T04:07:34.767 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2520.g69e2cd73.el 144/198 2024-03-29T04:07:34.798 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 145/198 2024-03-29T04:07:34.821 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 145/198 2024-03-29T04:07:34.822 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:34.822 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T04:07:34.822 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T04:07:34.822 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T04:07:34.822 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:34.865 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 146/198 2024-03-29T04:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 146/198 2024-03-29T04:07:34.926 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-2520.g69e2cd73.el9.x86_64 147/198 2024-03-29T04:07:35.804 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-29T04:07:35.834 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 149/198 2024-03-29T04:07:35.857 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 149/198 2024-03-29T04:07:35.857 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:35.857 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T04:07:35.857 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T04:07:35.857 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T04:07:35.857 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:35.893 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2520.g69e2c 150/198 2024-03-29T04:07:35.915 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2520.g69e2c 150/198 2024-03-29T04:07:35.915 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:35.915 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T04:07:35.915 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:35.960 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 151/198 2024-03-29T04:07:35.983 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 151/198 2024-03-29T04:07:35.983 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:35.983 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-29T04:07:35.983 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-29T04:07:35.983 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-29T04:07:35.983 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:36.303 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 152/198 2024-03-29T04:07:36.326 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 152/198 2024-03-29T04:07:36.326 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T04:07:36.326 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T04:07:36.326 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T04:07:36.326 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T04:07:36.327 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:37.661 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-2520.g69e2cd73.el9.x86_64 153/198 2024-03-29T04:07:37.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 154/198 2024-03-29T04:07:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 155/198 2024-03-29T04:07:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-29T04:07:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-2520.g69e2cd73.el9.x86_64 157/198 2024-03-29T04:07:38.069 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-29T04:07:38.109 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 159/198 2024-03-29T04:07:38.166 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-29T04:07:38.251 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-29T04:07:38.286 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-29T04:07:38.311 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-2520.g69e2cd73.el9.x86_ 163/198 2024-03-29T04:07:38.343 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-29T04:07:38.373 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-29T04:07:38.457 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-29T04:07:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-29T04:07:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-29T04:07:39.135 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-29T04:07:39.179 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-29T04:07:39.199 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-29T04:07:40.010 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-2520.g69e2cd73.el9.noarch 170/198 2024-03-29T04:07:40.084 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-29T04:07:40.124 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-29T04:07:40.210 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-29T04:07:40.238 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-29T04:07:40.263 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-29T04:07:40.288 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-29T04:07:40.315 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-29T04:07:40.315 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-29T04:07:40.403 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-29T04:07:40.432 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-29T04:07:40.432 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-29T04:07:40.517 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-29T04:07:40.543 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-29T04:07:40.577 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-29T04:07:40.610 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-29T04:07:40.635 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-29T04:07:40.670 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-29T04:07:40.708 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-29T04:07:40.741 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-29T04:07:40.766 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-29T04:07:40.794 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-29T04:07:40.817 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-29T04:07:40.832 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-2520.g69e2cd73.el9.x86_64 153/198 2024-03-29T04:07:40.842 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-29T04:07:40.849 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-29T04:07:40.849 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-29T04:07:40.873 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 154/198 2024-03-29T04:07:40.912 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 155/198 2024-03-29T04:07:40.948 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-29T04:07:41.029 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-29T04:07:41.029 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-29T04:07:41.112 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-29T04:07:41.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-2520.g69e2cd73.el9.x86_64 157/198 2024-03-29T04:07:41.136 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-29T04:07:41.168 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-29T04:07:41.196 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-29T04:07:41.221 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-29T04:07:41.258 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-29T04:07:41.288 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-29T04:07:41.305 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 159/198 2024-03-29T04:07:41.358 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-29T04:07:41.414 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-29T04:07:41.465 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-29T04:07:41.498 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-2520.g69e2cd73.el9.x86_ 163/198 2024-03-29T04:07:41.538 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-29T04:07:41.568 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-29T04:07:41.624 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-29T04:07:41.684 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-29T04:07:41.716 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-29T04:07:42.291 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-29T04:07:42.340 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-29T04:07:42.361 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-29T04:07:43.189 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-2520.g69e2cd73.el9.noarch 170/198 2024-03-29T04:07:43.266 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-29T04:07:43.304 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-29T04:07:43.386 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-29T04:07:43.417 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-29T04:07:43.442 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-29T04:07:43.467 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-29T04:07:43.494 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-29T04:07:43.494 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-29T04:07:43.565 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-29T04:07:43.603 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-29T04:07:43.603 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-29T04:07:43.688 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-29T04:07:43.714 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-29T04:07:43.748 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-29T04:07:43.773 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-29T04:07:43.798 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-29T04:07:43.833 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-29T04:07:43.896 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-29T04:07:43.928 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-29T04:07:43.953 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-29T04:07:43.979 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-29T04:07:44.012 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-29T04:07:44.046 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-29T04:07:44.052 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-29T04:07:44.052 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-29T04:07:44.241 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-29T04:07:44.241 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-29T04:07:44.307 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-29T04:07:44.332 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-29T04:07:44.364 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-29T04:07:44.392 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-29T04:07:44.416 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-29T04:07:44.483 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-29T04:07:46.955 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-29T04:07:46.955 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-2520.g69e2cd73.el9.x86_64 1/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 2/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 3/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 4/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2520.g69e2c 5/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 6/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 7/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 8/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 9/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 10/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 11/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-2520.g69e2cd73.el9.x86_64 12/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 13/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-2520.g69e2cd73.el9.x86_64 14/198 2024-03-29T04:07:46.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-2520.g69e2cd73.el9.x86_ 15/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 16/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 17/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-2520.g69e2cd73.el9.x86_6 18/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86 19/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 20/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2520.g69e2cd73.el 21/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-2520.g69e2cd73.el9. 22/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-2520.g69e2cd73.el9.x86_6 23/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-2520.g69e2cd73.el9.x86_64 24/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 25/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 26/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 27/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 28/198 2024-03-29T04:07:46.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 29/198 2024-03-29T04:07:46.973 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2520.g69e2cd73. 30/198 2024-03-29T04:07:46.973 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noa 31/198 2024-03-29T04:07:46.973 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.n 32/198 2024-03-29T04:07:46.973 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e 33/198 2024-03-29T04:07:46.973 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2520.g69e2cd73.el 34/198 2024-03-29T04:07:46.973 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 35/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2520.g69e2cd73.e 36/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 37/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 38/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-2520.g69e2cd73.el9.noarch 39/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-29T04:07:46.974 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-29T04:07:46.975 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-29T04:07:46.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-29T04:07:46.977 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-29T04:07:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-29T04:07:46.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-29T04:07:46.980 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-29T04:07:46.981 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 151/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 153/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-29T04:07:46.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-29T04:07:46.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-29T04:07:46.984 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-29T04:07:46.985 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-29T04:07:46.985 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-29T04:07:46.985 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-29T04:07:46.985 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-29T04:07:46.985 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-29T04:07:46.985 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-29T04:07:49.864 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-2520.g69e2cd73.el9.x86_64 1/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 2/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 3/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 4/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2520.g69e2c 5/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 6/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 7/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 8/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 9/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 10/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 11/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-2520.g69e2cd73.el9.x86_64 12/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 13/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-2520.g69e2cd73.el9.x86_64 14/198 2024-03-29T04:07:49.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-2520.g69e2cd73.el9.x86_ 15/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 16/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 17/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-2520.g69e2cd73.el9.x86_6 18/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86 19/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 20/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2520.g69e2cd73.el 21/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-2520.g69e2cd73.el9. 22/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-2520.g69e2cd73.el9.x86_6 23/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-2520.g69e2cd73.el9.x86_64 24/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 25/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 26/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 27/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 28/198 2024-03-29T04:07:49.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 29/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2520.g69e2cd73. 30/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noa 31/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.n 32/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e 33/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2520.g69e2cd73.el 34/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 35/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2520.g69e2cd73.e 36/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 37/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 38/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-2520.g69e2cd73.el9.noarch 39/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-29T04:07:49.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-29T04:07:49.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-29T04:07:49.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-29T04:07:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-29T04:07:49.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-29T04:07:49.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-29T04:07:49.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-29T04:07:49.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 151/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 153/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-29T04:07:49.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-29T04:07:49.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-29T04:07:49.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-29T04:07:49.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: acl-2.3.1-4.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-29T04:07:50.034 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.035 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.036 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.037 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T04:07:50.038 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T04:07:50.039 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.14-1.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-29T04:07:50.040 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T04:07:50.041 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T04:07:50.042 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T04:07:50.043 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T04:07:50.044 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.045 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:50.046 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T04:07:50.555 DEBUG:teuthology.parallel:result is None 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: acl-2.3.1-4.el9.x86_64 2024-03-29T04:07:52.883 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-29T04:07:52.884 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.885 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.886 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-2520.g69e2cd73.el9.noarch 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T04:07:52.887 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-16.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T04:07:52.888 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T04:07:52.889 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.14-1.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T04:07:52.890 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T04:07:52.891 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T04:07:52.892 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-29T04:07:52.893 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T04:07:52.894 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T04:07:52.898 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-2520.g69e2cd73.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T04:07:52.899 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-29T04:07:52.900 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:52.900 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-29T04:07:53.388 DEBUG:teuthology.parallel:result is None 2024-03-29T04:07:53.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=69e2cd73c52284dfd5096c66b60c6e6b26a3c447 2024-03-29T04:07:53.530 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T04:07:53.553 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-2520.g69e2cd73.el9 2024-03-29T04:07:53.553 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2520.g69e2cd73.el9 2024-03-29T04:07:53.553 INFO:teuthology.task.install:The correct ceph version 19.0.0-2520.g69e2cd73 is installed. 2024-03-29T04:07:53.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=69e2cd73c52284dfd5096c66b60c6e6b26a3c447 2024-03-29T04:07:53.772 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T04:07:53.799 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-2520.g69e2cd73.el9 2024-03-29T04:07:53.799 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2520.g69e2cd73.el9 2024-03-29T04:07:53.799 INFO:teuthology.task.install:The correct ceph version 19.0.0-2520.g69e2cd73 is installed. 2024-03-29T04:07:53.801 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T04:07:53.801 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:53.801 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T04:07:53.832 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:53.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T04:07:53.874 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T04:07:53.874 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:53.874 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T04:07:53.902 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T04:07:53.969 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:53.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T04:07:54.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T04:07:54.069 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T04:07:54.069 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:54.069 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T04:07:54.096 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T04:07:54.163 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:54.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T04:07:54.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T04:07:54.260 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T04:07:54.261 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:54.261 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T04:07:54.287 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T04:07:54.354 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:54.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T04:07:54.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T04:07:54.450 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T04:07:54.541 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T04:07:54.541 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2024-03-29T04:07:54.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-03-29T04:07:54.571 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T04:07:54.571 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T04:07:54.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T04:07:54.637 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T04:07:54.638 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T04:07:54.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T04:07:54.711 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-29T04:07:54.711 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-29T04:07:54.711 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-29T04:07:54.711 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:54.712 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-29T04:07:54.757 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-29T04:07:54.875 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-29T04:07:54.901 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:54.901 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-29T04:07:54.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-29T04:07:55.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-29T04:07:55.064 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T04:07:55.065 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-29T04:07:55.065 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080', 'branch': 'wip-mchangir-testing1-main-20240328-072412', 'description': 'fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7629080', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing1-main-20240328-072412'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing1-main-20240328-072412', 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447'}}, 'owner': 'scheduled_mchangir@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8207, 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447', 'sleep_before_teardown': 0, 'suite': 'fs:functional', 'suite_branch': 'wip-mchangir-testing1-main-20240328-072412', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvoTFsWWL4vZ7GzTmlpqlvDFCSFJM8MPUM/4cPygSQpM4a+eE5+Ilqcdje7rDFxqSiEeHkRd0vNIKMSl6fIEck=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2iheM/phZGpxb/nWh+qNqJPh21pUeEuzfV1bo+S1e/XORiFLwi+n9nvbrRFTnjMjUmEIh7WVJLrqjXkilT/Nk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '69e2cd73c52284dfd5096c66b60c6e6b26a3c447', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_uninlining']}}], 'teuthology': {'fragments_dropped': ['/cephfs/home/mchangir/src/git.ceph.com_ceph-c_wip-mchangir-testing1-main-20240328-072412/qa/suites/fs/functional/overrides/pg_health.yaml'], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-29_02:01:30', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T04:07:55.065 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T04:07:55.068 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T04:07:55.115 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T04:07:55.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T04:07:55.182 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:55.182 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T04:07:55.203 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T04:07:55.203 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-29 04:07:44.844483721 +0000 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-29 04:05:37.387617529 +0000 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-29 04:05:37.387617529 +0000 2024-03-29T04:07:55.259 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-29 04:05:37.387617529 +0000 2024-03-29T04:07:55.259 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T04:07:55.324 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-29T04:07:55.324 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-29T04:07:55.324 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000101951 s, 5.0 MB/s 2024-03-29T04:07:55.325 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T04:07:55.381 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-29 04:07:44.844483721 +0000 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-29 04:05:37.726611985 +0000 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-29 04:05:37.726611985 +0000 2024-03-29T04:07:55.437 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-29 04:05:37.726611985 +0000 2024-03-29T04:07:55.438 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T04:07:55.502 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-29T04:07:55.503 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-29T04:07:55.503 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000123449 s, 4.1 MB/s 2024-03-29T04:07:55.504 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T04:07:55.560 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-03-29T04:07:55.615 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T04:07:55.615 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:55.615 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T04:07:55.615 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:55.615 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:55.616 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-29 04:07:44.845483706 +0000 2024-03-29T04:07:55.616 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-29 04:05:38.053606637 +0000 2024-03-29T04:07:55.616 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-29 04:05:38.053606637 +0000 2024-03-29T04:07:55.616 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-29 04:05:38.053606637 +0000 2024-03-29T04:07:55.616 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T04:07:55.680 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-29T04:07:55.681 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-29T04:07:55.681 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000102599 s, 5.0 MB/s 2024-03-29T04:07:55.682 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T04:07:55.737 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-03-29T04:07:55.792 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T04:07:55.792 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:55.792 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T04:07:55.792 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:55.792 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:55.793 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-29 04:07:44.845483706 +0000 2024-03-29T04:07:55.793 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-29 04:05:38.383601240 +0000 2024-03-29T04:07:55.793 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-29 04:05:38.383601240 +0000 2024-03-29T04:07:55.793 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-29 04:05:38.383601240 +0000 2024-03-29T04:07:55.793 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T04:07:55.858 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-29T04:07:55.858 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-29T04:07:55.858 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000101478 s, 5.0 MB/s 2024-03-29T04:07:55.859 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T04:07:55.915 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T04:07:55.915 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:55.915 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T04:07:55.931 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T04:07:55.932 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-29T04:07:55.988 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T04:07:55.988 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:55.988 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T04:07:55.988 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:55.989 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:55.989 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 04:07:41.659020886 +0000 2024-03-29T04:07:55.989 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 04:05:40.048566501 +0000 2024-03-29T04:07:55.989 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 04:05:40.048566501 +0000 2024-03-29T04:07:55.989 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 04:05:40.048566501 +0000 2024-03-29T04:07:55.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T04:07:56.055 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T04:07:56.055 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T04:07:56.055 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000244985 s, 2.1 MB/s 2024-03-29T04:07:56.056 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T04:07:56.112 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-29T04:07:56.167 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T04:07:56.167 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:56.167 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T04:07:56.167 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:56.168 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:56.168 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 04:07:41.660020861 +0000 2024-03-29T04:07:56.168 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 04:05:40.393557737 +0000 2024-03-29T04:07:56.168 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 04:05:40.393557737 +0000 2024-03-29T04:07:56.168 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 04:05:40.393557737 +0000 2024-03-29T04:07:56.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T04:07:56.234 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T04:07:56.234 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T04:07:56.234 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000217184 s, 2.4 MB/s 2024-03-29T04:07:56.235 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T04:07:56.292 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-29T04:07:56.347 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T04:07:56.347 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:56.347 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T04:07:56.347 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:56.347 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:56.348 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 04:07:41.660020861 +0000 2024-03-29T04:07:56.348 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 04:05:40.732549126 +0000 2024-03-29T04:07:56.348 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 04:05:40.732549126 +0000 2024-03-29T04:07:56.348 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 04:05:40.732549126 +0000 2024-03-29T04:07:56.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T04:07:56.414 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T04:07:56.414 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T04:07:56.414 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000212496 s, 2.4 MB/s 2024-03-29T04:07:56.414 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T04:07:56.469 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 04:07:41.660020861 +0000 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 04:05:41.082540235 +0000 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 04:05:41.082540235 +0000 2024-03-29T04:07:56.524 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 04:05:41.082540235 +0000 2024-03-29T04:07:56.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T04:07:56.590 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T04:07:56.590 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T04:07:56.590 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000225294 s, 2.3 MB/s 2024-03-29T04:07:56.591 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T04:07:56.647 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T04:07:56.647 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T04:07:56.647 INFO:tasks.ceph:Generating config... 2024-03-29T04:07:56.648 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-29T04:07:56.648 INFO:tasks.ceph:[client] debug client = 20 2024-03-29T04:07:56.648 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T04:07:56.648 INFO:tasks.ceph:[client] pid file = 2024-03-29T04:07:56.649 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-29T04:07:56.649 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] mds_bal_replicate_threshold = 1 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T04:07:56.649 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T04:07:56.650 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T04:07:56.650 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T04:07:56.650 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T04:07:56.650 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-29T04:07:56.650 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T04:07:56.650 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-29T04:07:56.651 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T04:07:56.651 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T04:07:56.651 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T04:07:56.651 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T04:07:56.651 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T04:07:56.651 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-29T04:07:56.651 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T04:07:56.651 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T04:07:56.706 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T04:07:56.709 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T04:07:56.755 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T04:07:56.781 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.156'), ('mon.b', '172.21.15.203'), ('mon.c', '[v2:172.21.15.203:3301,v1:172.21.15.203:6790]')] 2024-03-29T04:07:56.782 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.156,172.21.15.203,[v2:172.21.15.203:3301,v1:172.21.15.203:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-29T04:07:56.783 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:56.783 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T04:07:56.836 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.156 --add b 172.21.15.203 --addv c '[v2:172.21.15.203:3301,v1:172.21.15.203:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:07:56.922 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: generated fsid 6a348a99-5402-4006-8e25-31481aaa0bf5 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:setting min_mon_release = quincy 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 0 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:fsid 6a348a99-5402-4006-8e25-31481aaa0bf5 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:last_changed 2024-03-29T04:07:56.944162+0000 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:created 2024-03-29T04:07:56.944162+0000 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:min_mon_release 17 (quincy) 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:election_strategy: 1 2024-03-29T04:07:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-03-29T04:07:56.924 INFO:teuthology.orchestra.run.smithi156.stdout:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-03-29T04:07:56.924 INFO:teuthology.orchestra.run.smithi156.stdout:2: [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] mon.c 2024-03-29T04:07:56.924 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T04:07:56.925 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T04:07:56.939 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6a348a99-5402-4006-8e25-31481aaa0bf5... 2024-03-29T04:07:56.940 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T04:07:56.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2024-03-29T04:07:57.030 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2024-03-29T04:07:57.031 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2024-03-29T04:07:57.032 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.156,172.21.15.203,[v2:172.21.15.203:3301,v1:172.21.15.203:6790]" 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = 6a348a99-5402-4006-8e25-31481aaa0bf5 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2024-03-29T04:07:57.033 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug trim objects = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2024-03-29T04:07:57.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: bdev async discard = True 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: bdev enable discard = True 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore allocator = bitmap 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore block size = 96636764160 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore fsck on mount = True 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluefs = 1/20 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluestore = 1/20 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2024-03-29T04:07:57.035 INFO:teuthology.orchestra.run.smithi156.stdout: debug rocksdb = 4/10 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd full ratio = 0.9 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd nearfull ratio = 0.8 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: osd failsafe full ratio = 0.95 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = bluestore 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2024-03-29T04:07:57.036 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: debug finisher = 20 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T04:07:57.037 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: mon down mkfs grace = 300 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2024-03-29T04:07:57.038 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = "" 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2024-03-29T04:07:57.039 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds balancer = 20 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: mds_bal_replicate_threshold = 1 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2024-03-29T04:07:57.040 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-03-29T04:07:57.041 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-03-29T04:07:57.042 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-03-29T04:07:57.043 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "172.21.15.156,172.21.15.203,[v2:172.21.15.203:3301,v1:172.21.15.203:6790]" 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 6a348a99-5402-4006-8e25-31481aaa0bf5 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-03-29T04:07:57.044 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.045 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: bdev async discard = True 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: bdev enable discard = True 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore allocator = bitmap 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 1/20 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 1/20 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 4/10 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-03-29T04:07:57.046 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: osd op complaint time = 180 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: debug client = 20 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: debug finisher = 20 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.047 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.048 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: mon down mkfs grace = 300 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: mon op complaint time = 120 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T04:07:57.049 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: client mount timeout = 600 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: debug client = 20 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = "" 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: rados mon op timeout = 900 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: rados osd op timeout = 900 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.a] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.c] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.b] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.d] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout:[mds] 2024-03-29T04:07:57.050 INFO:teuthology.orchestra.run.smithi203.stdout: debug mds = 20 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: debug mds balancer = 20 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: mds debug frag = True 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: mds debug scatterstat = True 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: mds op complaint time = 180 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: mds verify scatter = True 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: mds_bal_replicate_threshold = 1 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: osd op complaint time = 180 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: rados mon op timeout = 900 2024-03-29T04:07:57.051 INFO:teuthology.orchestra.run.smithi203.stdout: rados osd op timeout = 900 2024-03-29T04:07:57.052 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T04:07:57.053 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T04:07:57.148 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T04:07:57.148 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:57.148 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T04:07:57.162 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:57.162 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T04:07:57.217 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T04:07:57.296 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:07:57.296 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:07:57.351 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:07:57.352 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:57.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T04:07:57.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T04:07:57.392 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:07:57.392 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:07:57.447 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T04:07:57.447 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T04:07:57.447 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T04:07:57.502 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T04:07:57.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T04:07:57.582 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T04:07:57.584 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T04:07:57.585 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-29T04:07:57.639 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-29T04:07:57.642 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-29T04:07:57.668 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-29T04:07:57.763 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-29T04:07:57.766 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-29T04:07:57.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-29T04:07:57.848 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-29T04:07:57.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-29T04:07:57.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-29T04:07:57.972 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-29T04:07:57.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-29T04:07:58.001 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T04:07:58.001 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T04:07:58.044 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T04:07:58.059 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-29T04:07:58.142 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-29T04:07:58.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T04:07:58.199 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T04:07:58.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-29T04:07:58.297 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-29T04:07:58.312 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T04:07:58.312 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T04:07:58.312 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T04:07:58.339 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T04:07:58.339 INFO:tasks.ceph:role: osd.0 2024-03-29T04:07:58.339 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:07:58.340 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:07:58.406 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:07:58.439 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-29T04:07:58.441 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-29T04:07:58.441 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T04:07:58.512 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T04:07:58.584 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T04:07:58.652 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T04:07:58.652 INFO:tasks.ceph:role: osd.1 2024-03-29T04:07:58.652 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:07:58.652 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T04:07:58.718 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:07:58.719 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:07:58.750 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-29T04:07:58.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-29T04:07:58.752 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T04:07:58.823 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T04:07:58.894 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T04:07:58.961 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T04:07:58.961 INFO:tasks.ceph:role: osd.2 2024-03-29T04:07:58.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:07:58.962 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T04:07:59.027 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:07:59.027 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:07:59.028 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:07:59.060 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-29T04:07:59.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-29T04:07:59.062 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T04:07:59.132 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T04:07:59.203 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T04:07:59.271 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T04:07:59.271 INFO:tasks.ceph:role: osd.3 2024-03-29T04:07:59.271 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:07:59.271 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T04:07:59.339 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:07:59.339 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:07:59.339 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:07:59.340 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:07:59.372 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-29T04:07:59.374 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-29T04:07:59.374 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T04:07:59.445 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T04:07:59.517 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:07:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:07:59.629+0000 7fece0e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T04:07:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:07:59.629+0000 7fece0e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T04:07:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:07:59.629+0000 7fece0e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T04:07:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:07:59.629+0000 7fece0e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T04:07:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:07:59.629+0000 7fece0e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T04:07:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:07:59.629+0000 7fece0e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T04:08:02.464 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T04:08:02.495 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:02.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:02.605+0000 7f4832048640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T04:08:02.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:02.605+0000 7f4832048640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T04:08:02.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:02.605+0000 7f4832048640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T04:08:02.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:02.605+0000 7f4832048640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T04:08:02.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:02.605+0000 7f4832048640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T04:08:02.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:02.605+0000 7f4832048640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T04:08:05.434 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T04:08:05.465 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:05.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:05.578+0000 7f180fe0b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T04:08:05.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:05.578+0000 7f180fe0b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T04:08:05.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:05.578+0000 7f180fe0b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T04:08:05.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:05.578+0000 7f180fe0b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T04:08:05.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:05.578+0000 7f180fe0b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T04:08:05.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:05.578+0000 7f180fe0b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T04:08:08.402 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T04:08:08.435 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:08.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:08.548+0000 7f47e0de1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T04:08:08.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:08.548+0000 7f47e0de1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T04:08:08.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:08.548+0000 7f47e0de1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T04:08:08.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:08.548+0000 7f47e0de1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T04:08:08.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:08.548+0000 7f47e0de1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T04:08:08.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:08.549+0000 7f47e0de1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T04:08:11.368 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T04:08:11.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T04:08:11.432 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T04:08:11.432 INFO:tasks.ceph:role: osd.4 2024-03-29T04:08:11.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:08:11.432 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:08:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:08:11.988 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T04:08:11.990 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T04:08:11.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T04:08:12.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T04:08:12.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T04:08:12.234 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T04:08:12.234 INFO:tasks.ceph:role: osd.5 2024-03-29T04:08:12.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:08:12.234 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T04:08:12.308 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:08:12.309 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:08:12.771 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T04:08:12.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T04:08:12.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T04:08:12.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T04:08:12.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T04:08:12.963 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T04:08:12.964 INFO:tasks.ceph:role: osd.6 2024-03-29T04:08:12.964 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:08:12.964 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T04:08:13.039 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:08:13.039 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:08:13.039 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:08:13.039 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:08:13.039 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:08:13.040 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:08:13.040 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:08:13.040 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:08:13.040 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:08:13.040 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:08:13.514 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T04:08:13.515 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T04:08:13.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T04:08:13.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T04:08:13.670 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T04:08:13.737 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T04:08:13.737 INFO:tasks.ceph:role: osd.7 2024-03-29T04:08:13.737 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:08:13.737 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T04:08:13.812 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T04:08:14.309 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T04:08:14.310 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T04:08:14.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T04:08:14.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T04:08:14.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:14.614+0000 7fb9e3214640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T04:08:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:14.614+0000 7fb9e3214640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T04:08:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:14.614+0000 7fb9e3214640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T04:08:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:14.614+0000 7fb9e3214640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T04:08:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:14.614+0000 7fb9e3214640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T04:08:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:14.615+0000 7fb9e3214640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T04:08:17.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T04:08:17.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:17.596+0000 7f09ec78a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T04:08:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:17.596+0000 7f09ec78a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T04:08:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:17.596+0000 7f09ec78a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T04:08:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:17.596+0000 7f09ec78a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T04:08:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:17.596+0000 7f09ec78a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T04:08:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:17.597+0000 7f09ec78a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T04:08:20.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T04:08:20.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:20.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:20.564+0000 7f5789217640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T04:08:20.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:20.564+0000 7f5789217640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T04:08:20.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:20.564+0000 7f5789217640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T04:08:20.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:20.564+0000 7f5789217640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T04:08:20.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:20.564+0000 7f5789217640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T04:08:20.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:20.565+0000 7f5789217640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T04:08:23.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T04:08:23.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:23.525+0000 7f8d8272a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T04:08:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:23.525+0000 7f8d8272a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T04:08:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:23.525+0000 7f8d8272a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T04:08:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:23.525+0000 7f8d8272a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T04:08:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:23.525+0000 7f8d8272a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T04:08:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:23.526+0000 7f8d8272a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T04:08:25.067 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:08:25.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:08:26.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T04:08:26.389 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T04:08:26.389 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.389 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T04:08:26.422 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.422 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-29T04:08:26.492 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.492 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-29T04:08:26.562 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.562 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T04:08:26.631 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T04:08:26.699 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.700 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T04:08:26.768 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:26.768 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T04:08:26.836 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:26.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T04:08:26.868 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:26.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-29T04:08:26.940 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:26.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-29T04:08:27.011 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:27.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T04:08:27.080 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:27.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T04:08:27.149 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:27.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T04:08:27.218 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:27.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T04:08:27.286 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:27.286 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T04:08:27.303 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:27.303 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-29T04:08:27.360 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:27.360 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T04:08:27.375 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:27.375 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-29T04:08:27.430 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T04:08:27.430 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T04:08:27.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T04:08:27.473 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.y] 2024-03-29T04:08:27.473 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCdPgZm50w1HxAAw/V6YkFKxVtZwbu6zOW8kQ== 2024-03-29T04:08:27.473 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCdPgZmizBpJxAAE+UIT0JKBcd+kqBm8OXqJw== 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCdPgZmmynKLhAA381EIczFudxSPbO5dsgymw== 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCfPgZmWTSSJRAAMdLo56etlMy+UbuyTQxQ6g== 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCiPgZmvOckJBAA9sSR5OsAa3EEUuBvADm2AQ== 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQClPgZm1tyIIhAApvrDTh0Z96t6YlGcKltc2g== 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.3] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCoPgZmXJjCIBAA2Mew2foyDIQLWozE9B/rCg== 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2024-03-29T04:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCdPgZmEsv5IxAA7BsWzOOHkozAsY0Z+Jcwrg== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCdPgZmzh3TMxAAZG099AMS07J8m9Ou1su9Yg== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCdPgZmuIA3OxAAVTf4wVco/HrNU6fR69Nf/Q== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.4] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCuPgZmQh6oJBAAcylK7X8CI9jQn6Tq7v80hw== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.5] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCxPgZm3rGYIxAA690kx1Ijuzov0SoiOYtLjQ== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.6] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQC0PgZm5PWsIRAAO1bIBx89FMZwhLquciqsvg== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.7] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQC3PgZmLzxsHxAAJfLHVYR7HNXAqJav1bKs5g== 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-03-29T04:08:27.475 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCePgZmw7rsAxAA9fTOYwz/xRpNNRZVKRBg3w== 2024-03-29T04:08:27.476 INFO:teuthology.orchestra.run.smithi156.stdout:[client.2] 2024-03-29T04:08:27.476 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCePgZmwDS+CRAAW5ocjkKOFopCBTMtmRnsUQ== 2024-03-29T04:08:27.476 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2024-03-29T04:08:27.476 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCePgZmTQIwDRAAupi7/jFYwobM49T0Gl9F1g== 2024-03-29T04:08:27.476 INFO:teuthology.orchestra.run.smithi156.stdout:[client.3] 2024-03-29T04:08:27.476 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCePgZm0dIAExAAPMAWLLow0ngSQ0lNsDl9rA== 2024-03-29T04:08:27.495 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.y] 2024-03-29T04:08:27.495 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCdPgZm50w1HxAAw/V6YkFKxVtZwbu6zOW8kQ== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.a] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCdPgZmizBpJxAAE+UIT0JKBcd+kqBm8OXqJw== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.c] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCdPgZmmynKLhAA381EIczFudxSPbO5dsgymw== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCfPgZmWTSSJRAAMdLo56etlMy+UbuyTQxQ6g== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCiPgZmvOckJBAA9sSR5OsAa3EEUuBvADm2AQ== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQClPgZm1tyIIhAApvrDTh0Z96t6YlGcKltc2g== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCoPgZmXJjCIBAA2Mew2foyDIQLWozE9B/rCg== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCdPgZmEsv5IxAA7BsWzOOHkozAsY0Z+Jcwrg== 2024-03-29T04:08:27.496 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.b] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCdPgZmzh3TMxAAZG099AMS07J8m9Ou1su9Yg== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.d] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCdPgZmuIA3OxAAVTf4wVco/HrNU6fR69Nf/Q== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCuPgZmQh6oJBAAcylK7X8CI9jQn6Tq7v80hw== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCxPgZm3rGYIxAA690kx1Ijuzov0SoiOYtLjQ== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQC0PgZm5PWsIRAAO1bIBx89FMZwhLquciqsvg== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQC3PgZmLzxsHxAAJfLHVYR7HNXAqJav1bKs5g== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCePgZmw7rsAxAA9fTOYwz/xRpNNRZVKRBg3w== 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout:[client.2] 2024-03-29T04:08:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCePgZmwDS+CRAAW5ocjkKOFopCBTMtmRnsUQ== 2024-03-29T04:08:27.498 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-03-29T04:08:27.498 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCePgZmTQIwDRAAupi7/jFYwobM49T0Gl9F1g== 2024-03-29T04:08:27.498 INFO:teuthology.orchestra.run.smithi203.stdout:[client.3] 2024-03-29T04:08:27.498 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCePgZm0dIAExAAPMAWLLow0ngSQ0lNsDl9rA== 2024-03-29T04:08:27.499 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T04:08:27.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T04:08:27.593 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:27.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:27.650 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:27.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:27.707 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.762 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.817 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.871 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:27.925 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T04:08:27.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T04:08:27.979 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:27.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:28.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:28.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T04:08:28.094 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.149 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.206 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.260 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T04:08:28.314 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.371 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.429 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.486 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T04:08:28.542 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T04:08:28.543 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T04:08:28.569 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T04:08:28.668 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T04:08:28.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T04:08:28.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T04:08:28.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T04:08:28.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T04:08:28.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T04:08:29.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T04:08:29.040 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:29.042 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T04:08:29.095 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T04:08:29.095 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T04:08:29.095 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T04:08:29.097 INFO:tasks.ceph.mon.a:Started 2024-03-29T04:08:29.098 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T04:08:29.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T04:08:29.137 INFO:tasks.ceph.mon.b:Started 2024-03-29T04:08:29.138 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T04:08:29.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T04:08:29.139 INFO:tasks.ceph.mon.c:Started 2024-03-29T04:08:29.140 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T04:08:29.140 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T04:08:29.140 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T04:08:29.142 INFO:tasks.ceph.mgr.y:Started 2024-03-29T04:08:29.142 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T04:08:29.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T04:08:29.145 INFO:tasks.ceph.mgr.x:Started 2024-03-29T04:08:29.145 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T04:08:29.145 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T04:08:29.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.268+0000 7fb822ed9640 1 Processor -- start 2024-03-29T04:08:29.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.268+0000 7fb822ed9640 1 -- start start 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb822ed9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 0x7fb81c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb822ed9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb81c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb822ed9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb81c06e0b0 con 0x7fb81c0cccd0 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb822ed9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb81c0cd160 con 0x7fb81c156820 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb822ed9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb81c06e370 con 0x7fb81c0b6500 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 0x7fb81c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 0x7fb81c15d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48746/0 (socket says 172.21.15.156:48746) 2024-03-29T04:08:29.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 -- 172.21.15.156:0/1698983599 learned_addr learned my addr 172.21.15.156:0/1698983599 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 -- 172.21.15.156:0/1698983599 >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 msgr2=0x7fb81c15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 -- 172.21.15.156:0/1698983599 >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 msgr2=0x7fb81c15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 --2- 172.21.15.156:0/1698983599 >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 0x7fb81c15d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb81c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.269+0000 7fb820c75640 1 --2- 172.21.15.156:0/1698983599 >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 0x7fb81c15d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.270+0000 7fb821476640 1 -- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 msgr2=0x7fb81c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.270+0000 7fb821476640 1 -- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 msgr2=0x7fb81c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.270+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb81c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-29T04:08:29.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.270+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb81c0cd480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T04:08:29.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.431+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691497947 0 0) 0x7fb81c06e0b0 con 0x7fb81c0cccd0 2024-03-29T04:08:29.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.431+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb804003520 con 0x7fb81c0cccd0 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3561833494 0 0) 0x7fb804003520 con 0x7fb81c0cccd0 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 msgr2=0x7fb81c0cd480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb81c0cd480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 msgr2=0x7fb81c15d410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 --2- 172.21.15.156:0/1698983599 >> v2:172.21.15.156:3300/0 conn(0x7fb81c156820 0x7fb81c15d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81c06e650 con 0x7fb81c0cccd0 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fb810001670 con 0x7fb81c0cccd0 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.432+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 >> v1:172.21.15.203:6789/0 conn(0x7fb81c0cccd0 legacy=0x7fb81c0b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81b7fe640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb8040039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81b7fe640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb804003fe0 0x7fb804004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81b7fe640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb804004d40 0x7fb8040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c0cd160 con 0x7fb804004d40 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c06e370 con 0x7fb804003fe0 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb80400bac0 con 0x7fb81c0b6500 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb804004d40 0x7fb8040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb820c75640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb8040039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb81bfff640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb804003fe0 0x7fb804004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb821476640 1 -- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 msgr2=0x7fb8040039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c0b6500 0x7fb8040039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb821476640 1 -- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb804003fe0 msgr2=0x7fb804004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb804003fe0 0x7fb804004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.433+0000 7fb821476640 1 -- 172.21.15.156:0/1698983599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb804003520 con 0x7fb804004d40 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.434+0000 7fb821476640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb804004d40 0x7fb8040051c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb814009fc0 tx=0x7fb804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52af287f8890e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.434+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814039070 con 0x7fb804004d40 2024-03-29T04:08:29.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.434+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb81402cc70 con 0x7fb804004d40 2024-03-29T04:08:29.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.434+0000 7fb81b7fe640 1 -- 172.21.15.156:0/1698983599 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814009bc0 con 0x7fb804004d40 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb822ed9640 1 -- 172.21.15.156:0/1698983599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb804004d40 msgr2=0x7fb8040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb822ed9640 1 --2- 172.21.15.156:0/1698983599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb804004d40 0x7fb8040051c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb814009fc0 tx=0x7fb804005830 comp rx=0 tx=0).stop 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb821476640 1 -- 172.21.15.156:0/1698983599 reap_dead start 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb822ed9640 1 -- 172.21.15.156:0/1698983599 shutdown_connections 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb822ed9640 1 -- 172.21.15.156:0/1698983599 >> 172.21.15.156:0/1698983599 conn(0x7fb81c072ec0 msgr2=0x7fb81c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb822ed9640 1 -- 172.21.15.156:0/1698983599 shutdown_connections 2024-03-29T04:08:29.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.435+0000 7fb822ed9640 1 -- 172.21.15.156:0/1698983599 wait complete. 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.436+0000 7fb822ed9640 1 Processor -- start 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.436+0000 7fb822ed9640 1 -- start start 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb822ed9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb81c1517c0 0x7fb81c1f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb822ed9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c1f7ee0 0x7fb81c14bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb822ed9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 0x7fb81c140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb822ed9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c06ee50 con 0x7fb81c14c430 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb822ed9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c070730 con 0x7fb81c1517c0 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb822ed9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb81c06efd0 con 0x7fb81c1f7ee0 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 0x7fb81c140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb820c75640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb81c1517c0 0x7fb81c1f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 0x7fb81c140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48772/0 (socket says 172.21.15.156:48772) 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 -- 172.21.15.156:0/2561604832 learned_addr learned my addr 172.21.15.156:0/2561604832 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb81bfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c1f7ee0 0x7fb81c14bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 -- 172.21.15.156:0/2561604832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c1f7ee0 msgr2=0x7fb81c14bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c1f7ee0 0x7fb81c14bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 -- 172.21.15.156:0/2561604832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb81c1517c0 msgr2=0x7fb81c1f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb81c1517c0 0x7fb81c1f79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.437+0000 7fb821476640 1 -- 172.21.15.156:0/2561604832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81c151ba0 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb821476640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 0x7fb81c140070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb814007420 tx=0x7fb814004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814039040 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb814046070 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814009bc0 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81c14cc20 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb81c14d1e0 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb81403c070 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.438+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb81404bd20 con 0x7fb81c14c430 2024-03-29T04:08:29.421 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.439+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb81c073790 con 0x7fb81c14c430 2024-03-29T04:08:29.422 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.441+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb81402ce20 con 0x7fb81c14c430 2024-03-29T04:08:29.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.507+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb81c1f82c0 con 0x7fb81c14c430 2024-03-29T04:08:29.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.508+0000 7fb8197fa640 1 -- 172.21.15.156:0/2561604832 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb81404b460 con 0x7fb81c14c430 2024-03-29T04:08:29.488 INFO:teuthology.orchestra.run.smithi156.stderr:adjusted tunables profile to default 2024-03-29T04:08:29.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 msgr2=0x7fb81c140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:29.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 0x7fb81c140070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb814007420 tx=0x7fb814004ab0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 shutdown_connections 2024-03-29T04:08:29.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c14c430 0x7fb81c140070 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb81c1f7ee0 0x7fb81c14bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 --2- 172.21.15.156:0/2561604832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb81c1517c0 0x7fb81c1f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:29.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 >> 172.21.15.156:0/2561604832 conn(0x7fb81c072ec0 msgr2=0x7fb81c15e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:29.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.509+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 shutdown_connections 2024-03-29T04:08:29.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:29.510+0000 7fb822ed9640 1 -- 172.21.15.156:0/2561604832 wait complete. 2024-03-29T04:08:29.500 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T04:08:29.500 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T04:08:29.501 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:29.501 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T04:08:29.527 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:29.527 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T04:08:29.594 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:29.595 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T04:08:29.661 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:08:29.661 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T04:08:29.728 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:29.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T04:08:29.756 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:29.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T04:08:29.823 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:29.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T04:08:29.884 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:29.904+0000 7f7cb0a3f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T04:08:29.890 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:08:29.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T04:08:29.917 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:29.937+0000 7fe67f8fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T04:08:29.941 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:29.962+0000 7f7cb0a3f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T04:08:29.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 7f58abcd-38d8-4959-8eb1-927f56d35ad0 0 2024-03-29T04:08:29.970 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:29.990+0000 7fe67f8fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T04:08:30.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.119+0000 7f467c967640 1 Processor -- start 2024-03-29T04:08:30.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.119+0000 7f467c967640 1 -- start start 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f467c967640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f467815cb50 0x7f467815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f467c967640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46780cd940 0x7f4678160050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f467c967640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467815d470 con 0x7f46780cddc0 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f467c967640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f46780d01e0 con 0x7f467815cb50 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f467c967640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f46780d04c0 con 0x7f46780cd940 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f4675d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46780cd940 0x7f4678160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f4676d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f46780cddc0 0x7f4678160590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47038/0 (socket says 172.21.15.203:47038) 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f4675d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46780cd940 0x7f4678160050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57382/0 (socket says 172.21.15.203:57382) 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f4676d9d640 1 -- 172.21.15.203:0/1749108225 learned_addr learned my addr 172.21.15.203:0/1749108225 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.120+0000 7f467659c640 1 --2- 172.21.15.203:0/1749108225 >> v2:172.21.15.156:3300/0 conn(0x7f467815cb50 0x7f467815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f4675d9b640 1 -- 172.21.15.203:0/1749108225 >> v2:172.21.15.156:3300/0 conn(0x7f467815cb50 msgr2=0x7f467815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:30.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f4675d9b640 1 --2- 172.21.15.203:0/1749108225 >> v2:172.21.15.156:3300/0 conn(0x7f467815cb50 0x7f467815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f4675d9b640 1 -- 172.21.15.203:0/1749108225 >> v1:172.21.15.203:6789/0 conn(0x7f46780cddc0 legacy=0x7f4678160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f4675d9b640 1 -- 172.21.15.203:0/1749108225 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46780d07a0 con 0x7f46780cd940 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f467559a640 1 -- 172.21.15.203:0/1749108225 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388069900 0 0) 0x7f467815d470 con 0x7f46780cddc0 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f4675d9b640 1 --2- 172.21.15.203:0/1749108225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cd940 0x7f4678160050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f46780730f0 tx=0x7f466802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c464f0176c0c8596 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f467559a640 1 -- 172.21.15.203:0/1749108225 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668035020 con 0x7f46780cd940 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f467559a640 1 -- 172.21.15.203:0/1749108225 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4668031830 con 0x7f46780cd940 2024-03-29T04:08:30.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.121+0000 7f467559a640 1 -- 172.21.15.203:0/1749108225 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668031c00 con 0x7f46780cd940 2024-03-29T04:08:30.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 -- 172.21.15.203:0/1749108225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cd940 msgr2=0x7f4678160050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 --2- 172.21.15.203:0/1749108225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cd940 0x7f4678160050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f46780730f0 tx=0x7f466802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 -- 172.21.15.203:0/1749108225 shutdown_connections 2024-03-29T04:08:30.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 --2- 172.21.15.203:0/1749108225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cd940 0x7f4678160050 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 --2- 172.21.15.203:0/1749108225 >> v2:172.21.15.156:3300/0 conn(0x7f467815cb50 0x7f467815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 -- 172.21.15.203:0/1749108225 >> 172.21.15.203:0/1749108225 conn(0x7f46780b3d60 msgr2=0x7f46780b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 -- 172.21.15.203:0/1749108225 shutdown_connections 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.122+0000 7f467c967640 1 -- 172.21.15.203:0/1749108225 wait complete. 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 Processor -- start 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 -- start start 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46780cd940 0x7f46780b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 0x7f46780b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 0x7f46780c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46780cfe40 con 0x7f46780cd940 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46780d0d60 con 0x7f46780cddc0 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f467c967640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f46780cffc0 con 0x7f467815cb50 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f4675d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 0x7f46780b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.124+0000 7f4676d9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 0x7f46780c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 0x7f46780b6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57386/0 (socket says 172.21.15.203:57386) 2024-03-29T04:08:30.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 -- 172.21.15.203:0/701439377 learned_addr learned my addr 172.21.15.203:0/701439377 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4676d9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 0x7f46780c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56408/0 (socket says 172.21.15.203:56408) 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f467659c640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46780cd940 0x7f46780b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 -- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 msgr2=0x7f46780c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 0x7f46780c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 -- 172.21.15.203:0/701439377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46780cd940 msgr2=0x7f46780b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46780cd940 0x7f46780b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 -- 172.21.15.203:0/701439377 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46780b55a0 con 0x7f46780cddc0 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4676d9d640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 0x7f46780c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f467659c640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46780cd940 0x7f46780b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f4675d9b640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 0x7f46780b6640 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4668000c00 tx=0x7f46680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668035040 con 0x7f46780cddc0 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4668004230 con 0x7f46780cddc0 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466803a640 con 0x7f46780cddc0 2024-03-29T04:08:30.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46780b58e0 con 0x7f46780cddc0 2024-03-29T04:08:30.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.125+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46780b6120 con 0x7f46780cddc0 2024-03-29T04:08:30.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.126+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4668038020 con 0x7f46780cddc0 2024-03-29T04:08:30.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.126+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f4668003b50 con 0x7f46780cddc0 2024-03-29T04:08:30.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.127+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46780a03c0 con 0x7f46780cddc0 2024-03-29T04:08:30.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.129+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4668054080 con 0x7f46780cddc0 2024-03-29T04:08:30.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.218+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7f58abcd-38d8-4959-8eb1-927f56d35ad0", "id": 0} v 0) -- 0x7f467804f9d0 con 0x7f46780cddc0 2024-03-29T04:08:30.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.221+0000 7f466effd640 1 -- 172.21.15.203:0/701439377 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f58abcd-38d8-4959-8eb1-927f56d35ad0", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f46780b58e0 con 0x7f46780cddc0 2024-03-29T04:08:30.202 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-03-29T04:08:30.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.221+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 msgr2=0x7f46780b6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.221+0000 7f467c967640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 0x7f46780b6640 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4668000c00 tx=0x7f46680046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 shutdown_connections 2024-03-29T04:08:30.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f467815cb50 0x7f46780c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46780cddc0 0x7f46780b6640 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 --2- 172.21.15.203:0/701439377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46780cd940 0x7f46780b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 >> 172.21.15.203:0/701439377 conn(0x7f46780b3d60 msgr2=0x7f4678159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 shutdown_connections 2024-03-29T04:08:30.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.222+0000 7f467c967640 1 -- 172.21.15.203:0/701439377 wait complete. 2024-03-29T04:08:30.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 8eb20861-690f-4730-a707-16db70ce031e 1 2024-03-29T04:08:30.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.334+0000 7fd5b5a5b640 1 Processor -- start 2024-03-29T04:08:30.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.334+0000 7fd5b5a5b640 1 -- start start 2024-03-29T04:08:30.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5b5a5b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd5b015e540 0x7fd5b00cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5b5a5b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b00d2fd0 con 0x7fd5b00cfd60 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5b5a5b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b00d32b0 con 0x7fd5b00d2a00 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5b5a5b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b006c970 con 0x7fd5b015e540 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5ae7fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd5b00d2a00 0x7fd5b00cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47040/0 (socket says 172.21.15.203:47040) 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5ae7fc640 1 -- 172.21.15.203:0/4199535094 learned_addr learned my addr 172.21.15.203:0/4199535094 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161346373 0 0) 0x7fd5b00d32b0 con 0x7fd5b00d2a00 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5aeffd640 1 --2- 172.21.15.203:0/4199535094 >> v2:172.21.15.156:3300/0 conn(0x7fd5b015e540 0x7fd5b00cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd598003520 con 0x7fd5b00d2a00 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3761109546 0 0) 0x7fd598003520 con 0x7fd5b00d2a00 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 >> v2:172.21.15.156:3300/0 conn(0x7fd5b015e540 msgr2=0x7fd5b00cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 --2- 172.21.15.203:0/4199535094 >> v2:172.21.15.156:3300/0 conn(0x7fd5b015e540 0x7fd5b00cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 >> v1:172.21.15.156:6789/0 conn(0x7fd5b00cfd60 legacy=0x7fd5b006ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b006c600 con 0x7fd5b00d2a00 2024-03-29T04:08:30.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fd5a4001670 con 0x7fd5b00d2a00 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 >> v1:172.21.15.203:6789/0 conn(0x7fd5b00d2a00 legacy=0x7fd5b00cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5980039b0 0x7fd598003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd598004800 0x7fd598004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5980051c0 0x7fd59800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b006c970 con 0x7fd598004800 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd59800c090 con 0x7fd5980051c0 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.335+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd59800c290 con 0x7fd5980039b0 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5aeffd640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5980039b0 0x7fd598003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5ae7fc640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd598004800 0x7fd598004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5af7fe640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5980051c0 0x7fd59800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5aeffd640 1 -- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5980051c0 msgr2=0x7fd59800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5aeffd640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5980051c0 0x7fd59800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5aeffd640 1 -- 172.21.15.203:0/4199535094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd598004800 msgr2=0x7fd598004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5aeffd640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd598004800 0x7fd598004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5aeffd640 1 -- 172.21.15.203:0/4199535094 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd598003520 con 0x7fd5980039b0 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5af7fe640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5980051c0 0x7fd59800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.336+0000 7fd5af7fe640 1 -- 172.21.15.203:0/4199535094 reap_dead start 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5aeffd640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5980039b0 0x7fd598003df0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009af0 tx=0x7fd598005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=800690075d9b85e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0039070 con 0x7fd5980039b0 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a002c680 con 0x7fd5980039b0 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5adffb640 1 -- 172.21.15.203:0/4199535094 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0009690 con 0x7fd5980039b0 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5980039b0 msgr2=0x7fd598003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5980039b0 0x7fd598003df0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009af0 tx=0x7fd598005830 comp rx=0 tx=0).stop 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/4199535094 shutdown_connections 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 --2- 172.21.15.203:0/4199535094 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5980039b0 0x7fd598003df0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/4199535094 >> 172.21.15.203:0/4199535094 conn(0x7fd5b00d4450 msgr2=0x7fd5b00d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/4199535094 shutdown_connections 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.337+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/4199535094 wait complete. 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.338+0000 7fd5b5a5b640 1 Processor -- start 2024-03-29T04:08:30.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 -- start start 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5980039b0 0x7fd5b01613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 0x7fd5b0153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 0x7fd5b00ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00cf110 con 0x7fd5980039b0 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00cf290 con 0x7fd5b00b6300 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5b5a5b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b00cf410 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 0x7fd5b00ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5ae7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 0x7fd5b0153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 0x7fd5b00ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56434/0 (socket says 172.21.15.203:56434) 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5ae7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 0x7fd5b0153b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57396/0 (socket says 172.21.15.203:57396) 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5af7fe640 1 -- 172.21.15.203:0/2541968855 learned_addr learned my addr 172.21.15.203:0/2541968855 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.339+0000 7fd5aeffd640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5980039b0 0x7fd5b01613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5af7fe640 1 -- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 msgr2=0x7fd5b0153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5af7fe640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 0x7fd5b0153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5af7fe640 1 -- 172.21.15.203:0/2541968855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5980039b0 msgr2=0x7fd5b01613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5af7fe640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5980039b0 0x7fd5b01613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5af7fe640 1 -- 172.21.15.203:0/2541968855 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b00b6ac0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5ae7fc640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 0x7fd5b0153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5af7fe640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 0x7fd5b00ca940 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00ba500 tx=0x7fd5a800feb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8001a70 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a8018070 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b00c80d0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8003da0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b0154040 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5a80050d0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.340+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fd5a801d810 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.341+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b00a23a0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.343+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd5a80053c0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.416+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8eb20861-690f-4730-a707-16db70ce031e", "id": 1} v 0) -- 0x7fd5b004f9d0 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.419+0000 7fd597fff640 1 -- 172.21.15.203:0/2541968855 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8eb20861-690f-4730-a707-16db70ce031e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fd5a800d650 con 0x7fd5b00ca4c0 2024-03-29T04:08:30.399 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 msgr2=0x7fd5b00ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 0x7fd5b00ca940 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00ba500 tx=0x7fd5a800feb0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 shutdown_connections 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd5b00ca4c0 0x7fd5b00ca940 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b00b6300 0x7fd5b0153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 --2- 172.21.15.203:0/2541968855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5980039b0 0x7fd5b01613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 >> 172.21.15.203:0/2541968855 conn(0x7fd5b00d4450 msgr2=0x7fd5b00d4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 shutdown_connections 2024-03-29T04:08:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.420+0000 7fd5b5a5b640 1 -- 172.21.15.203:0/2541968855 wait complete. 2024-03-29T04:08:30.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 72345b8a-b884-4048-853a-520cd76b6ae6 2 2024-03-29T04:08:30.506 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.526+0000 7fe67f8fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T04:08:30.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.533+0000 7fd99311c640 1 Processor -- start 2024-03-29T04:08:30.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.533+0000 7fd99311c640 1 -- start start 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd99311c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd99311c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98c0b2b10 con 0x7fd98c0b40d0 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd99311c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98c0b5f20 con 0x7fd98c160460 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd99311c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c15efc0 con 0x7fd98c06b5b0 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd990eb8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd9916b9640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd98c160460 0x7fd98c168290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47048/0 (socket says 172.21.15.203:47048) 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd9916b9640 1 -- 172.21.15.203:0/3197441848 learned_addr learned my addr 172.21.15.203:0/3197441848 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd990eb8640 1 -- 172.21.15.203:0/3197441848 >> v1:172.21.15.203:6789/0 conn(0x7fd98c160460 legacy=0x7fd98c168290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd990eb8640 1 -- 172.21.15.203:0/3197441848 >> v1:172.21.15.156:6789/0 conn(0x7fd98c0b40d0 legacy=0x7fd98c06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd98b7fe640 1 -- 172.21.15.203:0/3197441848 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020261379 0 0) 0x7fd98c0b5f20 con 0x7fd98c160460 2024-03-29T04:08:30.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.534+0000 7fd990eb8640 1 -- 172.21.15.203:0/3197441848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0b6200 con 0x7fd98c06b5b0 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.535+0000 7fd990eb8640 1 --2- 172.21.15.203:0/3197441848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c0b25d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9800097d0 tx=0x7fd98002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b00698a3c206cd28 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.535+0000 7fd98b7fe640 1 -- 172.21.15.203:0/3197441848 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980035020 con 0x7fd98c06b5b0 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.535+0000 7fd98b7fe640 1 -- 172.21.15.203:0/3197441848 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd980031890 con 0x7fd98c06b5b0 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.535+0000 7fd98b7fe640 1 -- 172.21.15.203:0/3197441848 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980031c60 con 0x7fd98c06b5b0 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 -- 172.21.15.203:0/3197441848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 msgr2=0x7fd98c0b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 --2- 172.21.15.203:0/3197441848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c0b25d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9800097d0 tx=0x7fd98002b930 comp rx=0 tx=0).stop 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 -- 172.21.15.203:0/3197441848 shutdown_connections 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 --2- 172.21.15.203:0/3197441848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c0b25d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 -- 172.21.15.203:0/3197441848 >> 172.21.15.203:0/3197441848 conn(0x7fd98c0b65f0 msgr2=0x7fd98c0b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 -- 172.21.15.203:0/3197441848 shutdown_connections 2024-03-29T04:08:30.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.536+0000 7fd99311c640 1 -- 172.21.15.203:0/3197441848 wait complete. 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.537+0000 7fd99311c640 1 Processor -- start 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.537+0000 7fd99311c640 1 -- start start 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd99311c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd99311c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd98c0b40d0 0x7fd98c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd99311c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd98c160460 0x7fd98c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd99311c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0b5ce0 con 0x7fd98c160460 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd99311c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0b5b60 con 0x7fd98c0b40d0 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd99311c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c147520 con 0x7fd98c06b5b0 2024-03-29T04:08:30.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd990eb8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd98bfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd98c0b40d0 0x7fd98c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd990eb8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c140a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56452/0 (socket says 172.21.15.203:56452) 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd990eb8640 1 -- 172.21.15.203:0/1054643174 learned_addr learned my addr 172.21.15.203:0/1054643174 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.538+0000 7fd9916b9640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd98c160460 0x7fd98c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd98bfff640 1 -- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 msgr2=0x7fd98c140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd98bfff640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd98bfff640 1 -- 172.21.15.203:0/1054643174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd98c160460 msgr2=0x7fd98c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd98bfff640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd98c160460 0x7fd98c15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd98bfff640 1 -- 172.21.15.203:0/1054643174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0af360 con 0x7fd98c0b40d0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd990eb8640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd98bfff640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd98c0b40d0 0x7fd98c0aebf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd97800c390 tx=0x7fd97800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978017070 con 0x7fd98c0b40d0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9780052d0 con 0x7fd98c0b40d0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c0af640 con 0x7fd98c0b40d0 2024-03-29T04:08:30.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978003830 con 0x7fd98c0b40d0 2024-03-29T04:08:30.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c0afc00 con 0x7fd98c0b40d0 2024-03-29T04:08:30.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd97801a070 con 0x7fd98c0b40d0 2024-03-29T04:08:30.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.539+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fd978012420 con 0x7fd98c0b40d0 2024-03-29T04:08:30.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.540+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c09f430 con 0x7fd98c0b40d0 2024-03-29T04:08:30.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.544+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd978021070 con 0x7fd98c0b40d0 2024-03-29T04:08:30.536 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.556+0000 7f7cb0a3f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T04:08:30.560 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.581+0000 7fe67f8fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T04:08:30.593 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.613+0000 7f7cb0a3f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T04:08:30.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.617+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "72345b8a-b884-4048-853a-520cd76b6ae6", "id": 2} v 0) -- 0x7fd98c144a40 con 0x7fd98c0b40d0 2024-03-29T04:08:30.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.620+0000 7fd9897fa640 1 -- 172.21.15.203:0/1054643174 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72345b8a-b884-4048-853a-520cd76b6ae6", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fd978011590 con 0x7fd98c0b40d0 2024-03-29T04:08:30.600 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-03-29T04:08:30.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd98c0b40d0 msgr2=0x7fd98c0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd98c0b40d0 0x7fd98c0aebf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd97800c390 tx=0x7fd97800c850 comp rx=0 tx=0).stop 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 shutdown_connections 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd98c160460 0x7fd98c15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd98c0b40d0 0x7fd98c0aebf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 --2- 172.21.15.203:0/1054643174 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd98c06b5b0 0x7fd98c140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 >> 172.21.15.203:0/1054643174 conn(0x7fd98c0b65f0 msgr2=0x7fd98c0b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 shutdown_connections 2024-03-29T04:08:30.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.621+0000 7fd99311c640 1 -- 172.21.15.203:0/1054643174 wait complete. 2024-03-29T04:08:30.605 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.626+0000 7fe67f8fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T04:08:30.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 264ca621-8d00-4dd0-aaac-7bd0f502a666 3 2024-03-29T04:08:30.643 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.663+0000 7f7cb0a3f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T04:08:30.659 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.679+0000 7fe67f8fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T04:08:30.701 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.721+0000 7fe67f8fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T04:08:30.709 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.729+0000 7f7cb0a3f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T04:08:30.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.732+0000 7fca4539d640 1 Processor -- start 2024-03-29T04:08:30.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.732+0000 7fca4539d640 1 -- start start 2024-03-29T04:08:30.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca4539d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fca40145bf0 0x7fca400d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca4539d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fca4006f930 0x7fca4006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca4539d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca40145fd0 con 0x7fca4015c9e0 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca4539d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fca4015d500 con 0x7fca40145bf0 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca4539d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fca400d0a70 con 0x7fca4006f930 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fca4006f930 0x7fca4006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fca4006f930 0x7fca4006fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57406/0 (socket says 172.21.15.203:57406) 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 -- 172.21.15.203:0/1152084774 learned_addr learned my addr 172.21.15.203:0/1152084774 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3ffff640 1 --2- 172.21.15.203:0/1152084774 >> v2:172.21.15.156:3300/0 conn(0x7fca40145bf0 0x7fca400d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 -- 172.21.15.203:0/1152084774 >> v2:172.21.15.156:3300/0 conn(0x7fca40145bf0 msgr2=0x7fca400d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 --2- 172.21.15.203:0/1152084774 >> v2:172.21.15.156:3300/0 conn(0x7fca40145bf0 0x7fca400d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 -- 172.21.15.203:0/1152084774 >> v1:172.21.15.156:6789/0 conn(0x7fca4015c9e0 legacy=0x7fca4015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.733+0000 7fca3f7fe640 1 -- 172.21.15.203:0/1152084774 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca400d0cd0 con 0x7fca4006f930 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.734+0000 7fca3f7fe640 1 --2- 172.21.15.203:0/1152084774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4006f930 0x7fca4006fdc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca400cf880 tx=0x7fca3002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41406aaed33729e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.734+0000 7fca3effd640 1 -- 172.21.15.203:0/1152084774 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30034070 con 0x7fca4006f930 2024-03-29T04:08:30.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.734+0000 7fca3effd640 1 -- 172.21.15.203:0/1152084774 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca30030860 con 0x7fca4006f930 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.734+0000 7fca3effd640 1 -- 172.21.15.203:0/1152084774 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30030c30 con 0x7fca4006f930 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 -- 172.21.15.203:0/1152084774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4006f930 msgr2=0x7fca4006fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 --2- 172.21.15.203:0/1152084774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4006f930 0x7fca4006fdc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca400cf880 tx=0x7fca3002b9a0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 -- 172.21.15.203:0/1152084774 shutdown_connections 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 --2- 172.21.15.203:0/1152084774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4006f930 0x7fca4006fdc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 --2- 172.21.15.203:0/1152084774 >> v2:172.21.15.156:3300/0 conn(0x7fca40145bf0 0x7fca400d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 -- 172.21.15.203:0/1152084774 >> 172.21.15.203:0/1152084774 conn(0x7fca4006bbe0 msgr2=0x7fca400b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 -- 172.21.15.203:0/1152084774 shutdown_connections 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.735+0000 7fca4539d640 1 -- 172.21.15.203:0/1152084774 wait complete. 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.736+0000 7fca4539d640 1 Processor -- start 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.736+0000 7fca4539d640 1 -- start start 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca4539d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca4006f930 0x7fca400b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca4539d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 0x7fca400b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca4539d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 0x7fca4006d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca4539d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca400d1570 con 0x7fca4006f930 2024-03-29T04:08:30.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca4539d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fca400d1290 con 0x7fca4015c9e0 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca4539d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fca400b4fc0 con 0x7fca40145bf0 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca44b9c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 0x7fca4006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 0x7fca400b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 0x7fca400b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56466/0 (socket says 172.21.15.203:56466) 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca44b9c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 0x7fca4006d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57420/0 (socket says 172.21.15.203:57420) 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 -- 172.21.15.203:0/2836031569 learned_addr learned my addr 172.21.15.203:0/2836031569 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3ffff640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca4006f930 0x7fca400b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 -- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 msgr2=0x7fca4006d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 0x7fca4006d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 -- 172.21.15.203:0/2836031569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca4006f930 msgr2=0x7fca400b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca4006f930 0x7fca400b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3f7fe640 1 -- 172.21.15.203:0/2836031569 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca401f5350 con 0x7fca40145bf0 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca44b9c640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 0x7fca4006d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.737+0000 7fca3ffff640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca4006f930 0x7fca400b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca3f7fe640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 0x7fca400b36e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fca3002b9e0 tx=0x7fca30004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30048070 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca30030660 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30034040 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca400b3d10 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca401f4870 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fca3004b070 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.738+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fca300438a0 con 0x7fca40145bf0 2024-03-29T04:08:30.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.739+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca400d2710 con 0x7fca40145bf0 2024-03-29T04:08:30.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.741+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fca400d2710 con 0x7fca40145bf0 2024-03-29T04:08:30.755 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.775+0000 7f7cb0a3f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T04:08:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.812+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "264ca621-8d00-4dd0-aaac-7bd0f502a666", "id": 3} v 0) -- 0x7fca4004f8f0 con 0x7fca40145bf0 2024-03-29T04:08:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.815+0000 7fca3cff9640 1 -- 172.21.15.203:0/2836031569 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "264ca621-8d00-4dd0-aaac-7bd0f502a666", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fca30052070 con 0x7fca40145bf0 2024-03-29T04:08:30.796 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-03-29T04:08:30.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 msgr2=0x7fca400b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 0x7fca400b36e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fca3002b9e0 tx=0x7fca30004730 comp rx=0 tx=0).stop 2024-03-29T04:08:30.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 shutdown_connections 2024-03-29T04:08:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca4015c9e0 0x7fca4006d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca40145bf0 0x7fca400b36e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 --2- 172.21.15.203:0/2836031569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca4006f930 0x7fca400b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 >> 172.21.15.203:0/2836031569 conn(0x7fca4006bbe0 msgr2=0x7fca4015bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 shutdown_connections 2024-03-29T04:08:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.817+0000 7fca4539d640 1 -- 172.21.15.203:0/2836031569 wait complete. 2024-03-29T04:08:30.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new cb94351b-b0c4-4f09-9ce3-76657401e6bb 4 2024-03-29T04:08:30.819 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.839+0000 7fe67f8fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T04:08:30.884 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.904+0000 7f7cb0a3f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T04:08:30.889 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.910+0000 7fe67f8fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T04:08:30.935 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:30.956+0000 7fe67f8fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T04:08:30.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.973+0000 7f4443235640 1 Processor -- start 2024-03-29T04:08:30.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.973+0000 7f4443235640 1 -- start start 2024-03-29T04:08:30.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4443235640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c15ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4443235640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f443c0afff0 0x7f443c154b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4443235640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f443c0ade30 con 0x7f443c0b6970 2024-03-29T04:08:30.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4443235640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f443c0ae110 con 0x7f443c0afff0 2024-03-29T04:08:30.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4443235640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f443c0ae3f0 con 0x7f443c0b6590 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c15ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c15ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56480/0 (socket says 172.21.15.203:56480) 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 -- 172.21.15.203:0/4279554697 learned_addr learned my addr 172.21.15.203:0/4279554697 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f44417d2640 1 --2- 172.21.15.203:0/4279554697 >> v2:172.21.15.156:3300/0 conn(0x7f443c0afff0 0x7f443c154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 -- 172.21.15.203:0/4279554697 >> v2:172.21.15.156:3300/0 conn(0x7f443c0afff0 msgr2=0x7f443c154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 --2- 172.21.15.203:0/4279554697 >> v2:172.21.15.156:3300/0 conn(0x7f443c0afff0 0x7f443c154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 -- 172.21.15.203:0/4279554697 >> v1:172.21.15.156:6789/0 conn(0x7f443c0b6970 legacy=0x7f443c0af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f4440fd1640 1 -- 172.21.15.203:0/4279554697 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c0ae6d0 con 0x7f443c0b6590 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.974+0000 7f44417d2640 1 --2- 172.21.15.203:0/4279554697 >> v2:172.21.15.156:3300/0 conn(0x7f443c0afff0 0x7f443c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.975+0000 7f4440fd1640 1 --2- 172.21.15.203:0/4279554697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c15ea70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4430009870 tx=0x7f443002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=369826c323d1b8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.975+0000 7f443b7fe640 1 -- 172.21.15.203:0/4279554697 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430034070 con 0x7f443c0b6590 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.975+0000 7f443b7fe640 1 -- 172.21.15.203:0/4279554697 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4430030840 con 0x7f443c0b6590 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.975+0000 7f443b7fe640 1 -- 172.21.15.203:0/4279554697 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430030c10 con 0x7f443c0b6590 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 -- 172.21.15.203:0/4279554697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 msgr2=0x7f443c15ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 --2- 172.21.15.203:0/4279554697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c15ea70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4430009870 tx=0x7f443002b980 comp rx=0 tx=0).stop 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 -- 172.21.15.203:0/4279554697 shutdown_connections 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 --2- 172.21.15.203:0/4279554697 >> v2:172.21.15.156:3300/0 conn(0x7f443c0afff0 0x7f443c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 --2- 172.21.15.203:0/4279554697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c15ea70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 -- 172.21.15.203:0/4279554697 >> 172.21.15.203:0/4279554697 conn(0x7f443c072ec0 msgr2=0x7f443c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 -- 172.21.15.203:0/4279554697 shutdown_connections 2024-03-29T04:08:30.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.976+0000 7f4443235640 1 -- 172.21.15.203:0/4279554697 wait complete. 2024-03-29T04:08:30.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.977+0000 7f4443235640 1 Processor -- start 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.977+0000 7f4443235640 1 -- start start 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4443235640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 0x7f443c160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4443235640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4443235640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f443c0b6970 0x7f443c13fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4443235640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f443c070660 con 0x7f443c0b6970 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4443235640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f443c0adac0 con 0x7f443c0afff0 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4443235640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f443c0707e0 con 0x7f443c0b6590 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4440fd1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 0x7f443c160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f443bfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4440fd1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 0x7f443c160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57436/0 (socket says 172.21.15.203:57436) 2024-03-29T04:08:30.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4440fd1640 1 -- 172.21.15.203:0/1263295148 learned_addr learned my addr 172.21.15.203:0/1263295148 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:30.961 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f7cb0a3f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T04:08:30.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f44417d2640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f443c0b6970 0x7f443c13fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f443bfff640 1 -- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 msgr2=0x7f443c160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f443bfff640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 0x7f443c160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f443bfff640 1 -- 172.21.15.203:0/1263295148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f443c0b6970 msgr2=0x7f443c13fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f443bfff640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f443c0b6970 0x7f443c13fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f443bfff640 1 -- 172.21.15.203:0/1263295148 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c0cd4b0 con 0x7f443c0b6590 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.978+0000 7f4440fd1640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 0x7f443c160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f443bfff640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c161130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f442800c3c0 tx=0x7f442800c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44280052d0 con 0x7f443c0b6590 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4428005490 con 0x7f443c0b6590 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428007260 con 0x7f443c0b6590 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443c0cd790 con 0x7f443c0b6590 2024-03-29T04:08:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f443c0ce270 con 0x7f443c0b6590 2024-03-29T04:08:30.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4428007400 con 0x7f443c0b6590 2024-03-29T04:08:30.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.979+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4428011590 con 0x7f443c0b6590 2024-03-29T04:08:30.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.980+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f443c073790 con 0x7f443c0b6590 2024-03-29T04:08:30.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:30.982+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f44280039b0 con 0x7f443c0b6590 2024-03-29T04:08:30.988 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.008+0000 7fe67f8fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T04:08:31.007 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.027+0000 7f7cb0a3f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T04:08:31.035 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.056+0000 7fe67f8fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T04:08:31.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.055+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cb94351b-b0c4-4f09-9ce3-76657401e6bb", "id": 4} v 0) -- 0x7f443c15df00 con 0x7f443c0b6590 2024-03-29T04:08:31.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.059+0000 7f44397fa640 1 -- 172.21.15.203:0/1263295148 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb94351b-b0c4-4f09-9ce3-76657401e6bb", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4428003b90 con 0x7f443c0b6590 2024-03-29T04:08:31.039 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-03-29T04:08:31.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 msgr2=0x7f443c161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c161130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f442800c3c0 tx=0x7f442800c7c0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 shutdown_connections 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f443c0b6970 0x7f443c13fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f443c0b6590 0x7f443c161130 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 --2- 172.21.15.203:0/1263295148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f443c0afff0 0x7f443c160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 >> 172.21.15.203:0/1263295148 conn(0x7f443c072ec0 msgr2=0x7f443c0b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 shutdown_connections 2024-03-29T04:08:31.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.060+0000 7f4443235640 1 -- 172.21.15.203:0/1263295148 wait complete. 2024-03-29T04:08:31.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 6a37592b-2d57-4865-852b-50824052a319 5 2024-03-29T04:08:31.064 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.084+0000 7f7cb0a3f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T04:08:31.116 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.135+0000 7f7cb0a3f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T04:08:31.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.183+0000 7f618571c640 1 Processor -- start 2024-03-29T04:08:31.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f618571c640 1 -- start start 2024-03-29T04:08:31.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f618571c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f61800b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f618571c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f61800b4310 0x7f61800cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f618571c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618015cb10 con 0x7f61800cf340 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f618571c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f61800b4770 con 0x7f61800b4310 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f618571c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f618015cdf0 con 0x7f6180145c70 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.184+0000 7f617effd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f61800b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f61800cf340 0x7f618006f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47060/0 (socket says 172.21.15.203:47060) 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617f7fe640 1 -- 172.21.15.203:0/3145900606 learned_addr learned my addr 172.21.15.203:0/3145900606 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617e7fc640 1 --2- 172.21.15.203:0/3145900606 >> v2:172.21.15.156:3300/0 conn(0x7f61800b4310 0x7f61800cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617dffb640 1 -- 172.21.15.203:0/3145900606 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176692964 0 0) 0x7f618015cb10 con 0x7f61800cf340 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617dffb640 1 -- 172.21.15.203:0/3145900606 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6168003520 con 0x7f61800cf340 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617effd640 1 -- 172.21.15.203:0/3145900606 >> v2:172.21.15.156:3300/0 conn(0x7f61800b4310 msgr2=0x7f61800cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617effd640 1 --2- 172.21.15.203:0/3145900606 >> v2:172.21.15.156:3300/0 conn(0x7f61800b4310 0x7f61800cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617effd640 1 -- 172.21.15.203:0/3145900606 >> v1:172.21.15.203:6789/0 conn(0x7f61800cf340 legacy=0x7f618006f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.185+0000 7f617effd640 1 -- 172.21.15.203:0/3145900606 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618015d0d0 con 0x7f6180145c70 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f617effd640 1 --2- 172.21.15.203:0/3145900606 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f61800b3dd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f61740097a0 tx=0x7f617402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4367df15a32fc874 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f617dffb640 1 -- 172.21.15.203:0/3145900606 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174031780 con 0x7f6180145c70 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f617dffb640 1 -- 172.21.15.203:0/3145900606 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174037070 con 0x7f6180145c70 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f617dffb640 1 -- 172.21.15.203:0/3145900606 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174031b40 con 0x7f6180145c70 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 -- 172.21.15.203:0/3145900606 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 msgr2=0x7f61800b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 --2- 172.21.15.203:0/3145900606 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f61800b3dd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f61740097a0 tx=0x7f617402b820 comp rx=0 tx=0).stop 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 -- 172.21.15.203:0/3145900606 shutdown_connections 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 --2- 172.21.15.203:0/3145900606 >> v2:172.21.15.156:3300/0 conn(0x7f61800b4310 0x7f61800cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 --2- 172.21.15.203:0/3145900606 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f61800b3dd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 -- 172.21.15.203:0/3145900606 >> 172.21.15.203:0/3145900606 conn(0x7f61800b7660 msgr2=0x7f61800b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 -- 172.21.15.203:0/3145900606 shutdown_connections 2024-03-29T04:08:31.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.186+0000 7f618571c640 1 -- 172.21.15.203:0/3145900606 wait complete. 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 Processor -- start 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 -- start start 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61800b4310 0x7f61801f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61800cf340 0x7f61801f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f6180155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f61800cf930 con 0x7f61800cf340 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61800cf7b0 con 0x7f61800b4310 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.188+0000 7f618571c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6180146050 con 0x7f6180145c70 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61800b4310 0x7f61801f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f6180155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61800b4310 0x7f61801f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57444/0 (socket says 172.21.15.203:57444) 2024-03-29T04:08:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617effd640 1 -- 172.21.15.203:0/1457114342 learned_addr learned my addr 172.21.15.203:0/1457114342 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617e7fc640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61800cf340 0x7f61801f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 -- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61800b4310 msgr2=0x7f61801f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61800b4310 0x7f61801f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 -- 172.21.15.203:0/1457114342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61800cf340 msgr2=0x7f61801f5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61800cf340 0x7f61801f5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 -- 172.21.15.203:0/1457114342 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61801e6950 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f617f7fe640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f6180155d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f617800ca80 tx=0x7f617800cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178017070 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617800ac30 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61801e6c30 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178012600 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61801e71f0 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.189+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f617801a070 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.190+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f617801cd80 con 0x7f6180145c70 2024-03-29T04:08:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.190+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61800d02c0 con 0x7f6180145c70 2024-03-29T04:08:31.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.194+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f617801c800 con 0x7f6180145c70 2024-03-29T04:08:31.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.267+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6a37592b-2d57-4865-852b-50824052a319", "id": 5} v 0) -- 0x7f61800d2f60 con 0x7f6180145c70 2024-03-29T04:08:31.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.270+0000 7f61677fe640 1 -- 172.21.15.203:0/1457114342 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6a37592b-2d57-4865-852b-50824052a319", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f617801c9e0 con 0x7f6180145c70 2024-03-29T04:08:31.250 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-03-29T04:08:31.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 msgr2=0x7f6180155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f6180155d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f617800ca80 tx=0x7f617800cf40 comp rx=0 tx=0).stop 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 shutdown_connections 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6180145c70 0x7f6180155d00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61800cf340 0x7f61801f5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 --2- 172.21.15.203:0/1457114342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61800b4310 0x7f61801f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 >> 172.21.15.203:0/1457114342 conn(0x7f61800b7660 msgr2=0x7f61800b6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 shutdown_connections 2024-03-29T04:08:31.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.271+0000 7f618571c640 1 -- 172.21.15.203:0/1457114342 wait complete. 2024-03-29T04:08:31.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 30c85149-a723-4542-a1f9-e0745f60df69 6 2024-03-29T04:08:31.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.382+0000 7fd7b4435640 1 Processor -- start 2024-03-29T04:08:31.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.382+0000 7fd7b4435640 1 -- start start 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b4435640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd7ac15b7b0 0x7fd7ac0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b4435640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd7ac0b19e0 0x7fd7ac0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b4435640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ac0af440 con 0x7fd7ac0b4b70 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b4435640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac0af6f0 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b4435640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac0af9d0 con 0x7fd7ac15b7b0 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b19d0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd7ac15b7b0 0x7fd7ac0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b21d1640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd7ac0b4b70 0x7fd7ac0b4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47072/0 (socket says 172.21.15.203:47072) 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b21d1640 1 -- 172.21.15.203:0/3504984616 learned_addr learned my addr 172.21.15.203:0/3504984616 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b29d2640 1 --2- 172.21.15.203:0/3504984616 >> v2:172.21.15.156:3300/0 conn(0x7fd7ac0b19e0 0x7fd7ac0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b19d0640 1 -- 172.21.15.203:0/3504984616 >> v2:172.21.15.156:3300/0 conn(0x7fd7ac0b19e0 msgr2=0x7fd7ac0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:31.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b19d0640 1 --2- 172.21.15.203:0/3504984616 >> v2:172.21.15.156:3300/0 conn(0x7fd7ac0b19e0 0x7fd7ac0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b19d0640 1 -- 172.21.15.203:0/3504984616 >> v1:172.21.15.203:6789/0 conn(0x7fd7ac0b4b70 legacy=0x7fd7ac0b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b11cf640 1 -- 172.21.15.203:0/3504984616 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4138750395 0 0) 0x7fd7ac0af440 con 0x7fd7ac0b4b70 2024-03-29T04:08:31.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.383+0000 7fd7b19d0640 1 -- 172.21.15.203:0/3504984616 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac0b01d0 con 0x7fd7ac15b7b0 2024-03-29T04:08:31.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.384+0000 7fd7b19d0640 1 --2- 172.21.15.203:0/3504984616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac15b7b0 0x7fd7ac0ae9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0b2610 tx=0x7fd7a802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7806299e5915e6ad server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.384+0000 7fd7b11cf640 1 -- 172.21.15.203:0/3504984616 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a80315a0 con 0x7fd7ac15b7b0 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.384+0000 7fd7b11cf640 1 -- 172.21.15.203:0/3504984616 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a8037070 con 0x7fd7ac15b7b0 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.384+0000 7fd7b11cf640 1 -- 172.21.15.203:0/3504984616 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8031960 con 0x7fd7ac15b7b0 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 -- 172.21.15.203:0/3504984616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac15b7b0 msgr2=0x7fd7ac0ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 --2- 172.21.15.203:0/3504984616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac15b7b0 0x7fd7ac0ae9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0b2610 tx=0x7fd7a802b840 comp rx=0 tx=0).stop 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 -- 172.21.15.203:0/3504984616 shutdown_connections 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 --2- 172.21.15.203:0/3504984616 >> v2:172.21.15.156:3300/0 conn(0x7fd7ac0b19e0 0x7fd7ac0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 --2- 172.21.15.203:0/3504984616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac15b7b0 0x7fd7ac0ae9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 -- 172.21.15.203:0/3504984616 >> 172.21.15.203:0/3504984616 conn(0x7fd7ac0b65f0 msgr2=0x7fd7ac0b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 -- 172.21.15.203:0/3504984616 shutdown_connections 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.385+0000 7fd7b4435640 1 -- 172.21.15.203:0/3504984616 wait complete. 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.386+0000 7fd7b4435640 1 Processor -- start 2024-03-29T04:08:31.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.386+0000 7fd7b4435640 1 -- start start 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b4435640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 0x7fd7ac156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b4435640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7ac0b4b70 0x7fd7ac148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b4435640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7ac15b7b0 0x7fd7ac09b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b4435640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac06b940 con 0x7fd7ac0b4b70 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b4435640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac06b7c0 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b4435640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac06bac0 con 0x7fd7ac15b7b0 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 0x7fd7ac156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b29d2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7ac15b7b0 0x7fd7ac09b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 0x7fd7ac156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57462/0 (socket says 172.21.15.203:57462) 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 -- 172.21.15.203:0/4234042460 learned_addr learned my addr 172.21.15.203:0/4234042460 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b29d2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7ac15b7b0 0x7fd7ac09b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56510/0 (socket says 172.21.15.203:56510) 2024-03-29T04:08:31.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b19d0640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7ac0b4b70 0x7fd7ac148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 -- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7ac15b7b0 msgr2=0x7fd7ac09b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7ac15b7b0 0x7fd7ac09b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 -- 172.21.15.203:0/4234042460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7ac0b4b70 msgr2=0x7fd7ac148c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7ac0b4b70 0x7fd7ac148c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 -- 172.21.15.203:0/4234042460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac1491c0 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.387+0000 7fd7b21d1640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 0x7fd7ac156aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00c4d0 tx=0x7fd79c00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c018070 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c007500 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac157080 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c004a20 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac157640 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd79c01b070 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.388+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fd79c0129f0 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.389+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ac09f430 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.391+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd7ac1491c0 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.421 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.442+0000 7fe67f8fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T04:08:31.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.465+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "30c85149-a723-4542-a1f9-e0745f60df69", "id": 6} v 0) -- 0x7fd7ac04f9d0 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.468+0000 7fd7a2ffd640 1 -- 172.21.15.203:0/4234042460 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "30c85149-a723-4542-a1f9-e0745f60df69", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fd79c01d440 con 0x7fd7ac0b19e0 2024-03-29T04:08:31.448 INFO:teuthology.orchestra.run.smithi203.stdout:6 2024-03-29T04:08:31.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 msgr2=0x7fd7ac156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 0x7fd7ac156aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00c4d0 tx=0x7fd79c00c990 comp rx=0 tx=0).stop 2024-03-29T04:08:31.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 shutdown_connections 2024-03-29T04:08:31.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7ac15b7b0 0x7fd7ac09b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7ac0b4b70 0x7fd7ac148c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 --2- 172.21.15.203:0/4234042460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7ac0b19e0 0x7fd7ac156aa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 >> 172.21.15.203:0/4234042460 conn(0x7fd7ac0b65f0 msgr2=0x7fd7ac0b69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 shutdown_connections 2024-03-29T04:08:31.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.469+0000 7fd7b4435640 1 -- 172.21.15.203:0/4234042460 wait complete. 2024-03-29T04:08:31.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new f435f10c-bc5c-4df5-a966-310d860eb6d8 7 2024-03-29T04:08:31.526 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.546+0000 7f7cb0a3f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T04:08:31.556 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.577+0000 7fe67f8fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T04:08:31.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.581+0000 7ffb923e8640 1 Processor -- start 2024-03-29T04:08:31.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.581+0000 7ffb923e8640 1 -- start start 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb923e8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb8c06bcc0 0x7ffb8c0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb923e8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb923e8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb8c0b6370 con 0x7ffb8c161e40 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb923e8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ffb8c0b6120 con 0x7ffb8c06bcc0 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb923e8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ffb8c0b4190 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c0ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56524/0 (socket says 172.21.15.203:56524) 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 -- 172.21.15.203:0/2661001756 learned_addr learned my addr 172.21.15.203:0/2661001756 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb8b7fe640 1 --2- 172.21.15.203:0/2661001756 >> v2:172.21.15.156:3300/0 conn(0x7ffb8c06bcc0 0x7ffb8c0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 -- 172.21.15.203:0/2661001756 >> v2:172.21.15.156:3300/0 conn(0x7ffb8c06bcc0 msgr2=0x7ffb8c0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 --2- 172.21.15.203:0/2661001756 >> v2:172.21.15.156:3300/0 conn(0x7ffb8c06bcc0 0x7ffb8c0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 -- 172.21.15.203:0/2661001756 >> v1:172.21.15.156:6789/0 conn(0x7ffb8c161e40 legacy=0x7ffb8c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.582+0000 7ffb90985640 1 -- 172.21.15.203:0/2661001756 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8c0b4470 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.583+0000 7ffb90985640 1 --2- 172.21.15.203:0/2661001756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c0ae700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009e50 tx=0x7ffb8402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bea8ea1d8cae196 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.583+0000 7ffb8affd640 1 -- 172.21.15.203:0/2661001756 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8402c9e0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.583+0000 7ffb8affd640 1 -- 172.21.15.203:0/2661001756 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb8402cb80 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.583+0000 7ffb8affd640 1 -- 172.21.15.203:0/2661001756 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840343f0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 -- 172.21.15.203:0/2661001756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 msgr2=0x7ffb8c0ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 --2- 172.21.15.203:0/2661001756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c0ae700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009e50 tx=0x7ffb8402be40 comp rx=0 tx=0).stop 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 -- 172.21.15.203:0/2661001756 shutdown_connections 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 --2- 172.21.15.203:0/2661001756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c0ae700 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 --2- 172.21.15.203:0/2661001756 >> v2:172.21.15.156:3300/0 conn(0x7ffb8c06bcc0 0x7ffb8c0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 -- 172.21.15.203:0/2661001756 >> 172.21.15.203:0/2661001756 conn(0x7ffb8c0b6580 msgr2=0x7ffb8c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 -- 172.21.15.203:0/2661001756 shutdown_connections 2024-03-29T04:08:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.584+0000 7ffb923e8640 1 -- 172.21.15.203:0/2661001756 wait complete. 2024-03-29T04:08:31.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.585+0000 7ffb923e8640 1 Processor -- start 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.585+0000 7ffb923e8640 1 -- start start 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb923e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb8c06bcc0 0x7ffb8c147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb923e8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb923e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb8c161e40 0x7ffb8c15bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb923e8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c0aec40 con 0x7ffb8c161e40 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb923e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c0b1420 con 0x7ffb8c06bcc0 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb923e8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ffb8c0aedc0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c15b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:56536/0 (socket says 172.21.15.203:56536) 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 -- 172.21.15.203:0/1343153710 learned_addr learned my addr 172.21.15.203:0/1343153710 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8bfff640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb8c06bcc0 0x7ffb8c147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb90985640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb8c161e40 0x7ffb8c15bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 -- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb8c06bcc0 msgr2=0x7ffb8c147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb8c06bcc0 0x7ffb8c147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 -- 172.21.15.203:0/1343153710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb8c161e40 msgr2=0x7ffb8c15bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb8c161e40 0x7ffb8c15bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb8b7fe640 1 -- 172.21.15.203:0/1343153710 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8c15c4c0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.586+0000 7ffb90985640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb8c161e40 0x7ffb8c15bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb8b7fe640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c15b870 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e9e0 tx=0x7ffb7c00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c017070 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb7c00c9d0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8c15c7a0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c00cdc0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8c1478b0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffb7c01a070 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.587+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7ffb7c01cba0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.588+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8c09f3c0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.590+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ffb7c01c600 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.662+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f435f10c-bc5c-4df5-a966-310d860eb6d8", "id": 7} v 0) -- 0x7ffb8c04f9d0 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.667+0000 7ffb88ff9640 1 -- 172.21.15.203:0/1343153710 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f435f10c-bc5c-4df5-a966-310d860eb6d8", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7ffb7c01c910 con 0x7ffb8c0b25f0 2024-03-29T04:08:31.647 INFO:teuthology.orchestra.run.smithi203.stdout:7 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.667+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 msgr2=0x7ffb8c15b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.667+0000 7ffb923e8640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c15b870 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e9e0 tx=0x7ffb7c00eea0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 shutdown_connections 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb8c161e40 0x7ffb8c15bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffb8c0b25f0 0x7ffb8c15b870 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 --2- 172.21.15.203:0/1343153710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb8c06bcc0 0x7ffb8c147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 >> 172.21.15.203:0/1343153710 conn(0x7ffb8c0b6580 msgr2=0x7ffb8c0b6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 shutdown_connections 2024-03-29T04:08:31.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:08:31.668+0000 7ffb923e8640 1 -- 172.21.15.203:0/1343153710 wait complete. 2024-03-29T04:08:31.658 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T04:08:31.658 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T04:08:31.661 INFO:tasks.ceph.osd.0:Started 2024-03-29T04:08:31.661 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T04:08:31.661 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T04:08:31.665 INFO:tasks.ceph.osd.1:Started 2024-03-29T04:08:31.665 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T04:08:31.665 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T04:08:31.667 INFO:tasks.ceph.osd.2:Started 2024-03-29T04:08:31.667 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T04:08:31.667 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T04:08:31.669 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.689+0000 7f7cb0a3f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T04:08:31.669 INFO:tasks.ceph.osd.3:Started 2024-03-29T04:08:31.670 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T04:08:31.670 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T04:08:31.672 INFO:tasks.ceph.osd.4:Started 2024-03-29T04:08:31.672 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T04:08:31.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T04:08:31.674 INFO:tasks.ceph.osd.5:Started 2024-03-29T04:08:31.674 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T04:08:31.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T04:08:31.676 INFO:tasks.ceph.osd.6:Started 2024-03-29T04:08:31.677 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T04:08:31.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T04:08:31.679 INFO:tasks.ceph.osd.7:Started 2024-03-29T04:08:31.679 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:31.680 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.700+0000 7fe67f8fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T04:08:31.738 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.759+0000 7fe67f8fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T04:08:31.799 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.818+0000 7f7cb0a3f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T04:08:31.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.830+0000 7fe604e88640 1 Processor -- start 2024-03-29T04:08:31.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.830+0000 7fe604e88640 1 -- start start 2024-03-29T04:08:31.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe604e88640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b240 0x7fe6000b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe604e88640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe60015b620 0x7fe6000afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe604e88640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6000afa60 con 0x7fe6000b03d0 2024-03-29T04:08:31.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe604e88640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe6000b0810 con 0x7fe60015b620 2024-03-29T04:08:31.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe604e88640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe6000af4a0 con 0x7fe60015b240 2024-03-29T04:08:31.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fe59c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b240 0x7fe6000b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fdd9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe60015b620 0x7fe6000afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fe59c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b240 0x7fe6000b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:49616/0 (socket says 172.21.15.156:49616) 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fe59c640 1 -- 172.21.15.156:0/566174854 learned_addr learned my addr 172.21.15.156:0/566174854 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fdd9b640 1 -- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b240 msgr2=0x7fe6000b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fdd9b640 1 --2- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b240 0x7fe6000b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fdd9b640 1 -- 172.21.15.156:0/566174854 >> v1:172.21.15.203:6789/0 conn(0x7fe6000b03d0 legacy=0x7fe6000ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.831+0000 7fe5fdd9b640 1 -- 172.21.15.156:0/566174854 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6000af780 con 0x7fe60015b620 2024-03-29T04:08:31.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.832+0000 7fe5fdd9b640 1 --2- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe60015b620 0x7fe6000afe90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40097a0 tx=0x7fe5f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f30b31c8fbc0f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.832+0000 7fe5fd59a640 1 -- 172.21.15.156:0/566174854 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4035020 con 0x7fe60015b620 2024-03-29T04:08:31.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.832+0000 7fe5fd59a640 1 -- 172.21.15.156:0/566174854 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f4031830 con 0x7fe60015b620 2024-03-29T04:08:31.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.832+0000 7fe5fd59a640 1 -- 172.21.15.156:0/566174854 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4031c00 con 0x7fe60015b620 2024-03-29T04:08:31.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 -- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe60015b620 msgr2=0x7fe6000afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 --2- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe60015b620 0x7fe6000afe90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40097a0 tx=0x7fe5f402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 -- 172.21.15.156:0/566174854 shutdown_connections 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 --2- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe60015b620 0x7fe6000afe90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 --2- 172.21.15.156:0/566174854 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b240 0x7fe6000b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 -- 172.21.15.156:0/566174854 >> 172.21.15.156:0/566174854 conn(0x7fe60009aa40 msgr2=0x7fe6000b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 -- 172.21.15.156:0/566174854 shutdown_connections 2024-03-29T04:08:31.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.833+0000 7fe604e88640 1 -- 172.21.15.156:0/566174854 wait complete. 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.834+0000 7fe604e88640 1 Processor -- start 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.834+0000 7fe604e88640 1 -- start start 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe604e88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000b03d0 0x7fe60009d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe604e88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 0x7fe60009d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe604e88640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b620 0x7fe600149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe604e88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe6000aedc0 con 0x7fe60015b240 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe604e88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6000b43e0 con 0x7fe6000b03d0 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe604e88640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe6000aef40 con 0x7fe60015b620 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe5fdd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 0x7fe60009d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe5fe59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000b03d0 0x7fe60009d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe5fdd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 0x7fe60009d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43134/0 (socket says 172.21.15.156:43134) 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe5fdd9b640 1 -- 172.21.15.156:0/948680902 learned_addr learned my addr 172.21.15.156:0/948680902 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.835+0000 7fe5fed9d640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b620 0x7fe600149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fe59c640 1 -- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b620 msgr2=0x7fe600149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fe59c640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b620 0x7fe600149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fe59c640 1 -- 172.21.15.156:0/948680902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 msgr2=0x7fe60009d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fe59c640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 0x7fe60009d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fe59c640 1 -- 172.21.15.156:0/948680902 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6000d02a0 con 0x7fe6000b03d0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fdd9b640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 0x7fe60009d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5fe59c640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000b03d0 0x7fe60009d430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0010bc0 tx=0x7fe5f000b510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f001e070 con 0x7fe6000b03d0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f000be30 con 0x7fe6000b03d0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6000d0580 con 0x7fe6000b03d0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0019610 con 0x7fe6000b03d0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.836+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6000cd880 con 0x7fe6000b03d0 2024-03-29T04:08:31.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.837+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe5f0021070 con 0x7fe6000b03d0 2024-03-29T04:08:31.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.837+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe5f0022a80 con 0x7fe6000b03d0 2024-03-29T04:08:31.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.837+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6000ae6b0 con 0x7fe6000b03d0 2024-03-29T04:08:31.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.839+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe6000ae6b0 con 0x7fe6000b03d0 2024-03-29T04:08:31.846 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.866+0000 7f7cb0a3f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T04:08:31.865 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.886+0000 7fe67f8fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T04:08:31.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.900+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe60015aaa0 con 0x7fe6000b03d0 2024-03-29T04:08:31.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.901+0000 7fe5eeffd640 1 -- 172.21.15.156:0/948680902 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe60015aaa0 con 0x7fe6000b03d0 2024-03-29T04:08:31.880 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:31.880 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":10,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:31.665343+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000b03d0 msgr2=0x7fe60009d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000b03d0 0x7fe60009d430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0010bc0 tx=0x7fe5f000b510 comp rx=0 tx=0).stop 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 shutdown_connections 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe60015b620 0x7fe600149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60015b240 0x7fe60009d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 --2- 172.21.15.156:0/948680902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000b03d0 0x7fe60009d430 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 >> 172.21.15.156:0/948680902 conn(0x7fe60009aa40 msgr2=0x7fe60006f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 shutdown_connections 2024-03-29T04:08:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:31.902+0000 7fe604e88640 1 -- 172.21.15.156:0/948680902 wait complete. 2024-03-29T04:08:31.893 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T04:08:31.893 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T04:08:31.960 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:31.980+0000 7f7cb0a3f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T04:08:31.963 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:31.984+0000 7fe67f8fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T04:08:32.058 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:32.078+0000 7f7cb0a3f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T04:08:32.121 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:32.141+0000 7fe67f8fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T04:08:32.163 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:32.184+0000 7fe67f8fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T04:08:32.214 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:32.234+0000 7f7cb0a3f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T04:08:32.256 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:32.276+0000 7f7cb0a3f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T04:08:32.272 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-03-29T04:08:32.292+0000 7f24e8a17640 -1 Falling back to public interface 2024-03-29T04:08:32.277 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-29T04:08:32.297+0000 7fb2dfc0f640 -1 Falling back to public interface 2024-03-29T04:08:32.279 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-03-29T04:08:32.300+0000 7f31ebc24640 -1 Falling back to public interface 2024-03-29T04:08:32.282 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-29T04:08:32.302+0000 7f625918c640 -1 Falling back to public interface 2024-03-29T04:08:32.287 INFO:tasks.ceph.osd.3.smithi156.stderr:2024-03-29T04:08:32.308+0000 7fb9c28b5640 -1 Falling back to public interface 2024-03-29T04:08:32.290 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-29T04:08:32.310+0000 7efe51411640 -1 Falling back to public interface 2024-03-29T04:08:32.291 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-29T04:08:32.310+0000 7f303540e640 -1 Falling back to public interface 2024-03-29T04:08:32.294 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-03-29T04:08:32.315+0000 7fbfa800f640 -1 Falling back to public interface 2024-03-29T04:08:32.324 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:32.345+0000 7fe67f8fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T04:08:32.416 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:32.436+0000 7f7cb0a3f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T04:08:32.700 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:32.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.812+0000 7f791f0b1640 1 Processor -- start 2024-03-29T04:08:32.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.812+0000 7f791f0b1640 1 -- start start 2024-03-29T04:08:32.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f791f0b1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7918162250 0x7f79180d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:32.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f791f0b1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7918162c30 con 0x7f79181626d0 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f791f0b1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7918162f10 con 0x7f7918158da0 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f791f0b1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7918159180 con 0x7f7918162250 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f7917fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7918162250 0x7f79180d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f791d64e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f79181626d0 0x7f79180cef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:56312/0 (socket says 172.21.15.156:56312) 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f791d64e640 1 -- 172.21.15.156:0/3262550794 learned_addr learned my addr 172.21.15.156:0/3262550794 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f7917fff640 1 -- 172.21.15.156:0/3262550794 >> v1:172.21.15.203:6789/0 conn(0x7f7918158da0 legacy=0x7f79180d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f7917fff640 1 -- 172.21.15.156:0/3262550794 >> v1:172.21.15.156:6789/0 conn(0x7f79181626d0 legacy=0x7f79180cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.813+0000 7f7917fff640 1 -- 172.21.15.156:0/3262550794 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79180d0e90 con 0x7f7918162250 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.814+0000 7f7917fff640 1 --2- 172.21.15.156:0/3262550794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f79180d2510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f79040097a0 tx=0x7f790402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a3a07d246f05eb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.814+0000 7f79177fe640 1 -- 172.21.15.156:0/3262550794 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904037070 con 0x7f7918162250 2024-03-29T04:08:32.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.814+0000 7f79177fe640 1 -- 172.21.15.156:0/3262550794 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7904030710 con 0x7f7918162250 2024-03-29T04:08:32.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.814+0000 7f79177fe640 1 -- 172.21.15.156:0/3262550794 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904030ae0 con 0x7f7918162250 2024-03-29T04:08:32.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.814+0000 7f791f0b1640 1 -- 172.21.15.156:0/3262550794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 msgr2=0x7f79180d2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.814+0000 7f791f0b1640 1 --2- 172.21.15.156:0/3262550794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f79180d2510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f79040097a0 tx=0x7f790402b710 comp rx=0 tx=0).stop 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.815+0000 7f791f0b1640 1 -- 172.21.15.156:0/3262550794 shutdown_connections 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.815+0000 7f791f0b1640 1 --2- 172.21.15.156:0/3262550794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f79180d2510 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.815+0000 7f791f0b1640 1 -- 172.21.15.156:0/3262550794 >> 172.21.15.156:0/3262550794 conn(0x7f791809d790 msgr2=0x7f791808bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.815+0000 7f791f0b1640 1 -- 172.21.15.156:0/3262550794 shutdown_connections 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.815+0000 7f791f0b1640 1 -- 172.21.15.156:0/3262550794 wait complete. 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.816+0000 7f791f0b1640 1 Processor -- start 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.816+0000 7f791f0b1640 1 -- start start 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7918158da0 0x7f79180b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f791815bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79181626d0 0x7f7918152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:32.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79180b3590 con 0x7f7918162250 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79181631f0 con 0x7f79181626d0 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f79180b3710 con 0x7f7918158da0 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f791815bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f791815bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43138/0 (socket says 172.21.15.156:43138) 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 -- 172.21.15.156:0/1958469446 learned_addr learned my addr 172.21.15.156:0/1958469446 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791d64e640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79181626d0 0x7f7918152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791ce4d640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7918158da0 0x7f79180b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 -- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7918158da0 msgr2=0x7f79180b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7918158da0 0x7f79180b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 -- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79181626d0 msgr2=0x7f7918152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79181626d0 0x7f7918152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 -- 172.21.15.156:0/1958469446 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79180b4810 con 0x7f7918162250 2024-03-29T04:08:32.800 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f7917fff640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f791815bc80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7904000c00 tx=0x7f79040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904041020 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7904032a40 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904037040 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79180b4af0 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.817+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79180b50b0 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.818+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7904044020 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.818+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f790403cb40 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.818+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79180690c0 con 0x7f7918162250 2024-03-29T04:08:32.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.820+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f790404b030 con 0x7f7918162250 2024-03-29T04:08:32.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.881+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79180d5170 con 0x7f7918162250 2024-03-29T04:08:32.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.882+0000 7f79157fa640 1 -- 172.21.15.156:0/1958469446 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f790403cde0 con 0x7f7918162250 2024-03-29T04:08:32.862 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-29T04:08:32.862 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:31.665343+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:32.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 msgr2=0x7f791815bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:32.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f791815bc80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7904000c00 tx=0x7f79040046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 shutdown_connections 2024-03-29T04:08:32.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79181626d0 0x7f7918152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7918162250 0x7f791815bc80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 --2- 172.21.15.156:0/1958469446 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7918158da0 0x7f79180b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:32.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 >> 172.21.15.156:0/1958469446 conn(0x7f791809d790 msgr2=0x7f7918154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:32.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 shutdown_connections 2024-03-29T04:08:32.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:32.883+0000 7f791f0b1640 1 -- 172.21.15.156:0/1958469446 wait complete. 2024-03-29T04:08:32.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T04:08:33.162 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:33.183+0000 7fe67f8fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T04:08:33.237 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:33.257+0000 7f7cb0a3f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T04:08:33.439 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:33.459+0000 7fe67f8fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T04:08:33.489 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:33.510+0000 7fe67f8fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T04:08:33.518 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:33.538+0000 7f7cb0a3f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T04:08:33.570 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:33.590+0000 7f7cb0a3f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T04:08:33.604 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:33.624+0000 7fe67f8fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T04:08:33.697 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T04:08:33.716+0000 7f7cb0a3f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T04:08:34.619 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:34.640+0000 7fe620d1a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T04:08:34.620 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-29T04:08:34.640+0000 7fe620d1a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T04:08:35.744 INFO:tasks.ceph.osd.3.smithi156.stderr:2024-03-29T04:08:35.764+0000 7fb9c28b5640 -1 osd.3 0 log_to_monitors true 2024-03-29T04:08:35.796 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-03-29T04:08:35.816+0000 7f24e8a17640 -1 osd.1 0 log_to_monitors true 2024-03-29T04:08:35.797 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-29T04:08:35.816+0000 7efe51411640 -1 osd.4 0 log_to_monitors true 2024-03-29T04:08:35.804 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-29T04:08:35.824+0000 7fb2dfc0f640 -1 osd.6 0 log_to_monitors true 2024-03-29T04:08:35.804 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-03-29T04:08:35.825+0000 7f31ebc24640 -1 osd.0 0 log_to_monitors true 2024-03-29T04:08:35.811 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-03-29T04:08:35.831+0000 7fbfa800f640 -1 osd.2 0 log_to_monitors true 2024-03-29T04:08:35.816 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-29T04:08:35.836+0000 7f625918c640 -1 osd.5 0 log_to_monitors true 2024-03-29T04:08:35.818 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-29T04:08:35.838+0000 7f303540e640 -1 osd.7 0 log_to_monitors true 2024-03-29T04:08:38.924 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-03-29T04:08:38.944+0000 7f31e7e2e640 -1 osd.0 0 waiting for initial osdmap 2024-03-29T04:08:38.937 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-03-29T04:08:38.958+0000 7f31df43a640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:38.978 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-03-29T04:08:38.998+0000 7fbfa39af640 -1 osd.2 0 waiting for initial osdmap 2024-03-29T04:08:38.990 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-03-29T04:08:39.011+0000 7fbf9b7cf640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.056 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-03-29T04:08:39.077+0000 7f24e440c640 -1 osd.1 0 waiting for initial osdmap 2024-03-29T04:08:39.064 INFO:tasks.ceph.osd.3.smithi156.stderr:2024-03-29T04:08:39.085+0000 7fb9be84b640 -1 osd.3 0 waiting for initial osdmap 2024-03-29T04:08:39.066 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-03-29T04:08:39.087+0000 7f24dc22c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.074 INFO:tasks.ceph.osd.3.smithi156.stderr:2024-03-29T04:08:39.095+0000 7fb9b666b640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.678 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:39.711 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-29T04:08:39.731+0000 7f6255122640 -1 osd.5 0 waiting for initial osdmap 2024-03-29T04:08:39.717 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-29T04:08:39.738+0000 7efe4cdd3640 -1 osd.4 0 waiting for initial osdmap 2024-03-29T04:08:39.718 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-29T04:08:39.738+0000 7f3030e75640 -1 osd.7 0 waiting for initial osdmap 2024-03-29T04:08:39.729 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-29T04:08:39.750+0000 7f624cf42640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.735 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-29T04:08:39.756+0000 7f3028c95640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.738 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-29T04:08:39.759+0000 7efe44bf3640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.758 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-29T04:08:39.779+0000 7fb2db627640 -1 osd.6 0 waiting for initial osdmap 2024-03-29T04:08:39.770 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.790+0000 7f156feb1640 1 Processor -- start 2024-03-29T04:08:39.770 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.790+0000 7f156feb1640 1 -- start start 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156feb1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f156815e650 0x7f15680b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156feb1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f156806e170 0x7f156806c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156feb1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f15680b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156feb1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f156806cf70 con 0x7f156815e650 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156feb1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f156806f460 con 0x7f156806e170 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156feb1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f156806e5d0 con 0x7f156806cb00 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f156815e650 0x7f15680b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f156815e650 0x7f15680b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43258/0 (socket says 172.21.15.156:43258) 2024-03-29T04:08:39.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 -- 172.21.15.156:0/3063065144 learned_addr learned my addr 172.21.15.156:0/3063065144 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156e44e640 1 --2- 172.21.15.156:0/3063065144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f15680b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156d44c640 1 --2- 172.21.15.156:0/3063065144 >> v2:172.21.15.203:3300/0 conn(0x7f156806e170 0x7f156806c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 -- 172.21.15.156:0/3063065144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 msgr2=0x7f15680b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 --2- 172.21.15.156:0/3063065144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f15680b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 -- 172.21.15.156:0/3063065144 >> v2:172.21.15.203:3300/0 conn(0x7f156806e170 msgr2=0x7f156806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 --2- 172.21.15.156:0/3063065144 >> v2:172.21.15.203:3300/0 conn(0x7f156806e170 0x7f156806c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.791+0000 7f156dc4d640 1 -- 172.21.15.156:0/3063065144 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156806e850 con 0x7f156815e650 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.792+0000 7f156dc4d640 1 --2- 172.21.15.156:0/3063065144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156815e650 0x7f15680b5a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15680d0390 tx=0x7f156402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e8a2708c625258 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.792+0000 7f156cc4b640 1 -- 172.21.15.156:0/3063065144 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156402ca70 con 0x7f156815e650 2024-03-29T04:08:39.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.792+0000 7f156cc4b640 1 -- 172.21.15.156:0/3063065144 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f156402cc10 con 0x7f156815e650 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.792+0000 7f156cc4b640 1 -- 172.21.15.156:0/3063065144 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15640343f0 con 0x7f156815e650 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 -- 172.21.15.156:0/3063065144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156815e650 msgr2=0x7f15680b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 --2- 172.21.15.156:0/3063065144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156815e650 0x7f15680b5a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15680d0390 tx=0x7f156402bed0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 -- 172.21.15.156:0/3063065144 shutdown_connections 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 --2- 172.21.15.156:0/3063065144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f15680b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 --2- 172.21.15.156:0/3063065144 >> v2:172.21.15.203:3300/0 conn(0x7f156806e170 0x7f156806c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 --2- 172.21.15.156:0/3063065144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156815e650 0x7f15680b5a00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 -- 172.21.15.156:0/3063065144 >> 172.21.15.156:0/3063065144 conn(0x7f1568072110 msgr2=0x7f1568072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:39.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 -- 172.21.15.156:0/3063065144 shutdown_connections 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.793+0000 7f156feb1640 1 -- 172.21.15.156:0/3063065144 wait complete. 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.794+0000 7f156feb1640 1 Processor -- start 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.794+0000 7f156feb1640 1 -- start start 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156feb1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f156814d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156feb1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 0x7f15680c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156feb1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f156815e650 0x7f1568157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156feb1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f156806f0f0 con 0x7f156806e170 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156feb1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f156806ee10 con 0x7f156815e650 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156feb1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f15680b3e80 con 0x7f156806cb00 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 0x7f15680c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 0x7f15680c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43268/0 (socket says 172.21.15.156:43268) 2024-03-29T04:08:39.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 -- 172.21.15.156:0/3704059907 learned_addr learned my addr 172.21.15.156:0/3704059907 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156e44e640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f156815e650 0x7f1568157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156dc4d640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f156814d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 -- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 msgr2=0x7f156814d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f156814d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 -- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f156815e650 msgr2=0x7f1568157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f156815e650 0x7f1568157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 -- 172.21.15.156:0/3704059907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15680c61b0 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f156d44c640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 0x7f15680c58b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f155800e9b0 tx=0x7f155800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558017070 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.795+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155800c9a0 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.796+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155800cd90 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.796+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15680c6490 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.796+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15680c6a50 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.796+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156814aaa0 con 0x7f156806e170 2024-03-29T04:08:39.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.797+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f155801a070 con 0x7f156806e170 2024-03-29T04:08:39.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.797+0000 7f155e7fc640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f154c06c4a0 0x7f154c06e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:39.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.797+0000 7f156dc4d640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f154c06c4a0 0x7f154c06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:39.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.797+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3351+0+0 (secure 0 0 0) 0x7f155808afb0 con 0x7f156806e170 2024-03-29T04:08:39.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.798+0000 7f156dc4d640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f154c06c4a0 0x7f154c06e960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1564035000 tx=0x7f1564030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:39.782 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-29T04:08:39.794+0000 7fb2d3447640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T04:08:39.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.802+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f156814aaa0 con 0x7f156806e170 2024-03-29T04:08:39.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.977+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15680690c0 con 0x7f156806e170 2024-03-29T04:08:39.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.978+0000 7f155e7fc640 1 -- 172.21.15.156:0/3704059907 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+9001 (secure 0 0 0) 0x7f155805ee20 con 0x7f156806e170 2024-03-29T04:08:39.958 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-29T04:08:39.958 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":13,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:39.686376+0000","last_up_change":"2024-03-29T04:08:39.686376+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f154c06c4a0 msgr2=0x7f154c06e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f154c06c4a0 0x7f154c06e960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1564035000 tx=0x7f1564030000 comp rx=0 tx=0).stop 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 msgr2=0x7f15680c58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 0x7f15680c58b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f155800e9b0 tx=0x7f155800ee70 comp rx=0 tx=0).stop 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 shutdown_connections 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f154c06c4a0 0x7f154c06e960 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f156815e650 0x7f1568157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f156806e170 0x7f15680c58b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 --2- 172.21.15.156:0/3704059907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f156806cb00 0x7f156814d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:39.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 >> 172.21.15.156:0/3704059907 conn(0x7f1568072110 msgr2=0x7f156806c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:39.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 shutdown_connections 2024-03-29T04:08:39.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:39.980+0000 7f156feb1640 1 -- 172.21.15.156:0/3704059907 wait complete. 2024-03-29T04:08:39.969 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-03-29T04:08:46.777 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:46.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 Processor -- start 2024-03-29T04:08:46.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 -- start start 2024-03-29T04:08:46.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b2415e7d0 0x7f3b24162e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:46.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3b2415ebb0 0x7f3b240cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:46.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2415f430 con 0x7f3b240cf950 2024-03-29T04:08:46.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3b240cfdc0 con 0x7f3b2415ebb0 2024-03-29T04:08:46.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b29e74640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3b240d0f80 con 0x7f3b2415e7d0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.899+0000 7f3b22ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3b2415ebb0 0x7f3b240cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b22ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3b2415ebb0 0x7f3b240cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37964/0 (socket says 172.21.15.156:37964) 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b22ffd640 1 -- 172.21.15.156:0/3938081546 learned_addr learned my addr 172.21.15.156:0/3938081546 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b237fe640 1 --2- 172.21.15.156:0/3938081546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b2415e7d0 0x7f3b24162e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b22ffd640 1 -- 172.21.15.156:0/3938081546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b2415e7d0 msgr2=0x7f3b24162e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b22ffd640 1 --2- 172.21.15.156:0/3938081546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b2415e7d0 0x7f3b24162e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b22ffd640 1 -- 172.21.15.156:0/3938081546 >> v1:172.21.15.203:6789/0 conn(0x7f3b240cf950 legacy=0x7f3b240d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.900+0000 7f3b22ffd640 1 -- 172.21.15.156:0/3938081546 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2415f710 con 0x7f3b2415ebb0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b22ffd640 1 --2- 172.21.15.156:0/3938081546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415ebb0 0x7f3b240cf410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51601a86b35bd872 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b227fc640 1 -- 172.21.15.156:0/3938081546 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18035020 con 0x7f3b2415ebb0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b227fc640 1 -- 172.21.15.156:0/3938081546 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b18031640 con 0x7f3b2415ebb0 2024-03-29T04:08:46.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b227fc640 1 -- 172.21.15.156:0/3938081546 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1802d040 con 0x7f3b2415ebb0 2024-03-29T04:08:46.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 -- 172.21.15.156:0/3938081546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415ebb0 msgr2=0x7f3b240cf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 --2- 172.21.15.156:0/3938081546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415ebb0 0x7f3b240cf410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1802b8f0 comp rx=0 tx=0).stop 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 -- 172.21.15.156:0/3938081546 shutdown_connections 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 --2- 172.21.15.156:0/3938081546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415ebb0 0x7f3b240cf410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 --2- 172.21.15.156:0/3938081546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b2415e7d0 0x7f3b24162e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 -- 172.21.15.156:0/3938081546 >> 172.21.15.156:0/3938081546 conn(0x7f3b2408bf10 msgr2=0x7f3b2409d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 -- 172.21.15.156:0/3938081546 shutdown_connections 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.901+0000 7f3b29e74640 1 -- 172.21.15.156:0/3938081546 wait complete. 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.903+0000 7f3b29e74640 1 Processor -- start 2024-03-29T04:08:46.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.903+0000 7f3b29e74640 1 -- start start 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b240cf950 0x7f3b240b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 0x7f3b240b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b2415ebb0 0x7f3b2415c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2406c7f0 con 0x7f3b2415e7d0 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241633a0 con 0x7f3b2415ebb0 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2415fcd0 con 0x7f3b240cf950 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 0x7f3b240b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 0x7f3b240b8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37970/0 (socket says 172.21.15.156:37970) 2024-03-29T04:08:46.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 -- 172.21.15.156:0/657042835 learned_addr learned my addr 172.21.15.156:0/657042835 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b237fe640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b240cf950 0x7f3b240b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b23fff640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b2415ebb0 0x7f3b2415c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 -- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b240cf950 msgr2=0x7f3b240b65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b240cf950 0x7f3b240b65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 -- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b2415ebb0 msgr2=0x7f3b2415c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b2415ebb0 0x7f3b2415c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 -- 172.21.15.156:0/657042835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b240b8610 con 0x7f3b2415e7d0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b23fff640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b2415ebb0 0x7f3b2415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b22ffd640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 0x7f3b240b8cf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b18000c00 tx=0x7f3b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18040070 con 0x7f3b2415e7d0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b180399f0 con 0x7f3b2415e7d0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18035040 con 0x7f3b2415e7d0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.904+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b240b73a0 con 0x7f3b2415e7d0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.905+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b240b7960 con 0x7f3b2415e7d0 2024-03-29T04:08:46.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.906+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b240690c0 con 0x7f3b2415e7d0 2024-03-29T04:08:46.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.906+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3b1804b060 con 0x7f3b2415e7d0 2024-03-29T04:08:46.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.906+0000 7f3b0bfff640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3b1006c530 0x7f3b1006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:46.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.906+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f3b180acbb0 con 0x7f3b2415e7d0 2024-03-29T04:08:46.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.906+0000 7f3b237fe640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3b1006c530 0x7f3b1006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:46.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.907+0000 7f3b237fe640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3b1006c530 0x7f3b1006e9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3b140073f0 tx=0x7f3b1400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:46.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:46.911+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3b240b8610 con 0x7f3b2415e7d0 2024-03-29T04:08:47.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.089+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b2406c5d0 con 0x7f3b2415e7d0 2024-03-29T04:08:47.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.090+0000 7f3b0bfff640 1 -- 172.21.15.156:0/657042835 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7f3b1807b520 con 0x7f3b2415e7d0 2024-03-29T04:08:47.070 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-29T04:08:47.070 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":17,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:43.774034+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3b1006c530 msgr2=0x7f3b1006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3b1006c530 0x7f3b1006e9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3b140073f0 tx=0x7f3b1400b040 comp rx=0 tx=0).stop 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 msgr2=0x7f3b240b8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 0x7f3b240b8cf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b18000c00 tx=0x7f3b180046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 shutdown_connections 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3b1006c530 0x7f3b1006e9f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b2415ebb0 0x7f3b2415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b2415e7d0 0x7f3b240b8cf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 --2- 172.21.15.156:0/657042835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3b240cf950 0x7f3b240b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 >> 172.21.15.156:0/657042835 conn(0x7f3b2408bf10 msgr2=0x7f3b24067410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 shutdown_connections 2024-03-29T04:08:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-29T04:08:47.092+0000 7f3b29e74640 1 -- 172.21.15.156:0/657042835 wait complete. 2024-03-29T04:08:47.081 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-29T04:08:47.082 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-29T04:08:47.082 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T04:08:47.082 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T04:08:47.085 INFO:tasks.ceph.mds.a:Started 2024-03-29T04:08:47.085 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T04:08:47.085 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T04:08:47.087 INFO:tasks.ceph.mds.c:Started 2024-03-29T04:08:47.087 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T04:08:47.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T04:08:47.090 INFO:tasks.ceph.mds.b:Started 2024-03-29T04:08:47.090 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T04:08:47.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T04:08:47.092 INFO:tasks.ceph.mds.d:Started 2024-03-29T04:08:47.092 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-29T04:08:47.093 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:47.160 INFO:tasks.ceph.mds.a.smithi156.stdout:starting mds.a at 2024-03-29T04:08:47.162 INFO:tasks.ceph.mds.c.smithi156.stdout:starting mds.c at 2024-03-29T04:08:47.176 INFO:tasks.ceph.mds.b.smithi203.stdout:starting mds.b at 2024-03-29T04:08:47.177 INFO:tasks.ceph.mds.d.smithi203.stdout:starting mds.d at 2024-03-29T04:08:47.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.220+0000 7f7bf051c640 1 Processor -- start 2024-03-29T04:08:47.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.220+0000 7f7bf051c640 1 -- start start 2024-03-29T04:08:47.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bf051c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7be80cf450 0x7f7be80cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bf051c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be80d5140 0x7f7be80d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bf051c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be806caf0 con 0x7f7be80d17e0 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bf051c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7be80cf290 con 0x7f7be80cf450 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bf051c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7be80cfd70 con 0x7f7be80d5140 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7be80cf450 0x7f7be80cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7be80cf450 0x7f7be80cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38018/0 (socket says 172.21.15.156:38018) 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 -- 172.21.15.156:0/2002445940 learned_addr learned my addr 172.21.15.156:0/2002445940 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7beeab9640 1 --2- 172.21.15.156:0/2002445940 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be80d5140 0x7f7be80d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 -- 172.21.15.156:0/2002445940 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be80d5140 msgr2=0x7f7be80d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 --2- 172.21.15.156:0/2002445940 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be80d5140 0x7f7be80d1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 -- 172.21.15.156:0/2002445940 >> v1:172.21.15.203:6789/0 conn(0x7f7be80d17e0 legacy=0x7f7be806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.221+0000 7f7bee2b8640 1 -- 172.21.15.156:0/2002445940 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be80cefb0 con 0x7f7be80cf450 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.222+0000 7f7bee2b8640 1 --2- 172.21.15.156:0/2002445940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be80cf830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009870 tx=0x7f7bd802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b23552b8ce4ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.222+0000 7f7bed2b6640 1 -- 172.21.15.156:0/2002445940 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8034070 con 0x7f7be80cf450 2024-03-29T04:08:47.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.222+0000 7f7bed2b6640 1 -- 172.21.15.156:0/2002445940 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd8030840 con 0x7f7be80cf450 2024-03-29T04:08:47.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.222+0000 7f7bed2b6640 1 -- 172.21.15.156:0/2002445940 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8030c10 con 0x7f7be80cf450 2024-03-29T04:08:47.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.222+0000 7f7bf051c640 1 -- 172.21.15.156:0/2002445940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 msgr2=0x7f7be80cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.222+0000 7f7bf051c640 1 --2- 172.21.15.156:0/2002445940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be80cf830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009870 tx=0x7f7bd802b980 comp rx=0 tx=0).stop 2024-03-29T04:08:47.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.223+0000 7f7bf051c640 1 -- 172.21.15.156:0/2002445940 shutdown_connections 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.223+0000 7f7bf051c640 1 --2- 172.21.15.156:0/2002445940 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be80d5140 0x7f7be80d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.223+0000 7f7bf051c640 1 --2- 172.21.15.156:0/2002445940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be80cf830 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.223+0000 7f7bf051c640 1 -- 172.21.15.156:0/2002445940 >> 172.21.15.156:0/2002445940 conn(0x7f7be80d4410 msgr2=0x7f7be80d4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.223+0000 7f7bf051c640 1 -- 172.21.15.156:0/2002445940 shutdown_connections 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.223+0000 7f7bf051c640 1 -- 172.21.15.156:0/2002445940 wait complete. 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 Processor -- start 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 -- start start 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be8158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7be80d5140 0x7f7be80ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be81f4350 0x7f7be80b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7be806d330 con 0x7f7be80cf450 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7be806d050 con 0x7f7be80d5140 2024-03-29T04:08:47.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.224+0000 7f7bf051c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7be815e820 con 0x7f7be81f4350 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be8158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be8158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38034/0 (socket says 172.21.15.156:38034) 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 -- 172.21.15.156:0/773475584 learned_addr learned my addr 172.21.15.156:0/773475584 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bedab7640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7be80d5140 0x7f7be80ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7beeab9640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be81f4350 0x7f7be80b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 -- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be81f4350 msgr2=0x7f7be80b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be81f4350 0x7f7be80b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 -- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7be80d5140 msgr2=0x7f7be80ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7be80d5140 0x7f7be80ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 -- 172.21.15.156:0/773475584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be8159510 con 0x7f7be80cf450 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bee2b8640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be8158fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8000c00 tx=0x7f7bd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8048070 con 0x7f7be80cf450 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd8030640 con 0x7f7be80cf450 2024-03-29T04:08:47.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8034040 con 0x7f7be80cf450 2024-03-29T04:08:47.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be80cd830 con 0x7f7be80cf450 2024-03-29T04:08:47.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.225+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be80cddf0 con 0x7f7be80cf450 2024-03-29T04:08:47.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.226+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be8068610 con 0x7f7be80cf450 2024-03-29T04:08:47.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.227+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7bd804b070 con 0x7f7be80cf450 2024-03-29T04:08:47.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.227+0000 7f7bdeffd640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7bcc06c530 0x7f7bcc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.227+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7bd80ad2b0 con 0x7f7be80cf450 2024-03-29T04:08:47.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.227+0000 7f7bedab7640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7bcc06c530 0x7f7bcc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.227+0000 7f7bedab7640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7bcc06c530 0x7f7bcc06e9f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7be80cce70 tx=0x7f7be4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.231+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7be8159510 con 0x7f7be80cf450 2024-03-29T04:08:47.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.407+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7be80b6cd0 con 0x7f7be80cf450 2024-03-29T04:08:47.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.408+0000 7f7bdeffd640 1 -- 172.21.15.156:0/773475584 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7f7bd807b540 con 0x7f7be80cf450 2024-03-29T04:08:47.387 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:47.387 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:43.774034+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:47.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7bcc06c530 msgr2=0x7f7bcc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7bcc06c530 0x7f7bcc06e9f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7be80cce70 tx=0x7f7be4009040 comp rx=0 tx=0).stop 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 msgr2=0x7f7be8158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be8158fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8000c00 tx=0x7f7bd8004730 comp rx=0 tx=0).stop 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 shutdown_connections 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7bcc06c530 0x7f7bcc06e9f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7be81f4350 0x7f7be80b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7be80d5140 0x7f7be80ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 --2- 172.21.15.156:0/773475584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be80cf450 0x7f7be8158fd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 >> 172.21.15.156:0/773475584 conn(0x7f7be80d4410 msgr2=0x7f7be80b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 shutdown_connections 2024-03-29T04:08:47.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.410+0000 7f7bf051c640 1 -- 172.21.15.156:0/773475584 wait complete. 2024-03-29T04:08:47.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:08:47.401 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:08:47.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.524+0000 7f31189af640 1 Processor -- start 2024-03-29T04:08:47.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.524+0000 7f31189af640 1 -- start start 2024-03-29T04:08:47.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f31189af640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f311415e920 0x7f31140d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f31189af640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f31140cfc30 0x7f31140d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f31189af640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31140cf190 con 0x7f31140d2120 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f31189af640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f31140cf7a0 con 0x7f31140cfc30 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f31189af640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f31140d3080 con 0x7f311415e920 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f3112d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f31140cfc30 0x7f31140d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f3111d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f31140d2120 0x7f311406c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48754/0 (socket says 172.21.15.156:48754) 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.525+0000 7f3111d9b640 1 -- 172.21.15.156:0/2682643707 learned_addr learned my addr 172.21.15.156:0/2682643707 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f311259c640 1 --2- 172.21.15.156:0/2682643707 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f311415e920 0x7f31140d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f3112d9d640 1 -- 172.21.15.156:0/2682643707 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f311415e920 msgr2=0x7f31140d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f3112d9d640 1 --2- 172.21.15.156:0/2682643707 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f311415e920 0x7f31140d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f3112d9d640 1 -- 172.21.15.156:0/2682643707 >> v1:172.21.15.156:6789/0 conn(0x7f31140d2120 legacy=0x7f311406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f3112d9d640 1 -- 172.21.15.156:0/2682643707 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31140cf430 con 0x7f31140cfc30 2024-03-29T04:08:47.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f311159a640 1 -- 172.21.15.156:0/2682643707 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 795953293 0 0) 0x7f31140cf190 con 0x7f31140d2120 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.526+0000 7f3112d9d640 1 --2- 172.21.15.156:0/2682643707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31140cfc30 0x7f31140d00e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f310c0097a0 tx=0x7f310c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef972226982d5a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f311159a640 1 -- 172.21.15.156:0/2682643707 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c035020 con 0x7f31140cfc30 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f311159a640 1 -- 172.21.15.156:0/2682643707 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f310c031830 con 0x7f31140cfc30 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f311159a640 1 -- 172.21.15.156:0/2682643707 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c031c00 con 0x7f31140cfc30 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 -- 172.21.15.156:0/2682643707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31140cfc30 msgr2=0x7f31140d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 --2- 172.21.15.156:0/2682643707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31140cfc30 0x7f31140d00e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f310c0097a0 tx=0x7f310c02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 -- 172.21.15.156:0/2682643707 shutdown_connections 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 --2- 172.21.15.156:0/2682643707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31140cfc30 0x7f31140d00e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 --2- 172.21.15.156:0/2682643707 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f311415e920 0x7f31140d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 -- 172.21.15.156:0/2682643707 >> 172.21.15.156:0/2682643707 conn(0x7f311409d7a0 msgr2=0x7f31140d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:47.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 -- 172.21.15.156:0/2682643707 shutdown_connections 2024-03-29T04:08:47.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.527+0000 7f31189af640 1 -- 172.21.15.156:0/2682643707 wait complete. 2024-03-29T04:08:47.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.528+0000 7f31189af640 1 Processor -- start 2024-03-29T04:08:47.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 -- start start 2024-03-29T04:08:47.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31140cfc30 0x7f311415a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f31140d2120 0x7f31140b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 0x7f31140b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f31140d34e0 con 0x7f311415e920 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31140d3360 con 0x7f31140cfc30 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f31189af640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f31140d2c20 con 0x7f31140d2120 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f3112d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 0x7f31140b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f3112d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 0x7f31140b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38054/0 (socket says 172.21.15.156:38054) 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f3112d9d640 1 -- 172.21.15.156:0/2781844622 learned_addr learned my addr 172.21.15.156:0/2781844622 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f3111d9b640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f31140d2120 0x7f31140b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.529+0000 7f311259c640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31140cfc30 0x7f311415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3112d9d640 1 -- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f31140d2120 msgr2=0x7f31140b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3112d9d640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f31140d2120 0x7f31140b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3112d9d640 1 -- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31140cfc30 msgr2=0x7f311415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3112d9d640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31140cfc30 0x7f311415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3112d9d640 1 -- 172.21.15.156:0/2781844622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31140b9490 con 0x7f311415e920 2024-03-29T04:08:47.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3112d9d640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 0x7f31140b2150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f310c02bee0 tx=0x7f310c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c03b940 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f310c03bae0 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31141f83b0 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c0453f0 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.530+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31141f8970 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.531+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3114161070 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.531+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f310c038020 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.532+0000 7f3102ffd640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f30f406c530 0x7f30f406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.532+0000 7f311259c640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f30f406c530 0x7f30f406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.532+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f310c0acf30 con 0x7f311415e920 2024-03-29T04:08:47.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.532+0000 7f311259c640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f30f406c530 0x7f30f406e9f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3104003aa0 tx=0x7f3104009830 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.536+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f310c040070 con 0x7f311415e920 2024-03-29T04:08:47.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.711+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3114068fc0 con 0x7f311415e920 2024-03-29T04:08:47.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.711+0000 7f3102ffd640 1 -- 172.21.15.156:0/2781844622 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f3114068fc0 con 0x7f311415e920 2024-03-29T04:08:47.691 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f30f406c530 msgr2=0x7f30f406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f30f406c530 0x7f30f406e9f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3104003aa0 tx=0x7f3104009830 comp rx=0 tx=0).stop 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 msgr2=0x7f31140b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 0x7f31140b2150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f310c02bee0 tx=0x7f310c0046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 shutdown_connections 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f30f406c530 0x7f30f406e9f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f311415e920 0x7f31140b2150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f31140d2120 0x7f31140b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 --2- 172.21.15.156:0/2781844622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31140cfc30 0x7f311415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 >> 172.21.15.156:0/2781844622 conn(0x7f311409d7a0 msgr2=0x7f31140d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:47.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 shutdown_connections 2024-03-29T04:08:47.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.713+0000 7f31189af640 1 -- 172.21.15.156:0/2781844622 wait complete. 2024-03-29T04:08:47.704 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:08:47.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.825+0000 7fc6bfe8f640 1 Processor -- start 2024-03-29T04:08:47.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.825+0000 7fc6bfe8f640 1 -- start start 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bfe8f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 0x7fc6b8160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bfe8f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 0x7fc6b809b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bfe8f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b806fc80 con 0x7fc6b8162230 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bfe8f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b81627a0 con 0x7fc6b8070ad0 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bfe8f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b8162a80 con 0x7fc6b815c4f0 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6be42c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 0x7fc6b809b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bdc2b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 0x7fc6b8160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bdc2b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 0x7fc6b8160030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:56672/0 (socket says 172.21.15.156:56672) 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6be42c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 0x7fc6b809b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:33362/0 (socket says 172.21.15.156:33362) 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.826+0000 7fc6bdc2b640 1 -- 172.21.15.156:0/3104299761 learned_addr learned my addr 172.21.15.156:0/3104299761 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.827+0000 7fc6bdc2b640 1 -- 172.21.15.156:0/3104299761 >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 msgr2=0x7fc6b809b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.827+0000 7fc6bdc2b640 1 --2- 172.21.15.156:0/3104299761 >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 0x7fc6b809b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.827+0000 7fc6bdc2b640 1 -- 172.21.15.156:0/3104299761 >> v1:172.21.15.203:6789/0 conn(0x7fc6b8162230 legacy=0x7fc6b80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.827+0000 7fc6bdc2b640 1 -- 172.21.15.156:0/3104299761 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b8162d60 con 0x7fc6b815c4f0 2024-03-29T04:08:47.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.827+0000 7fc6be42c640 1 --2- 172.21.15.156:0/3104299761 >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 0x7fc6b809b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:47.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.827+0000 7fc6bdc2b640 1 --2- 172.21.15.156:0/3104299761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 0x7fc6b8160030 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009870 tx=0x7fc6b402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98be344b7bc0288b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bcc29640 1 -- 172.21.15.156:0/3104299761 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4034070 con 0x7fc6b815c4f0 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bcc29640 1 -- 172.21.15.156:0/3104299761 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b4030840 con 0x7fc6b815c4f0 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bcc29640 1 -- 172.21.15.156:0/3104299761 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4030c10 con 0x7fc6b815c4f0 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/3104299761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 msgr2=0x7fc6b8160030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/3104299761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 0x7fc6b8160030 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009870 tx=0x7fc6b402b980 comp rx=0 tx=0).stop 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/3104299761 shutdown_connections 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/3104299761 >> v2:172.21.15.203:3300/0 conn(0x7fc6b8070ad0 0x7fc6b809b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/3104299761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b815c4f0 0x7fc6b8160030 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/3104299761 >> 172.21.15.156:0/3104299761 conn(0x7fc6b80d2250 msgr2=0x7fc6b80d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/3104299761 shutdown_connections 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.828+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/3104299761 wait complete. 2024-03-29T04:08:47.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.829+0000 7fc6bfe8f640 1 Processor -- start 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.829+0000 7fc6bfe8f640 1 -- start start 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bfe8f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b8070ad0 0x7fc6b8153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bfe8f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 0x7fc6b806c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bfe8f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b8162230 0x7fc6b80cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bfe8f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b80d05a0 con 0x7fc6b815c4f0 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bfe8f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b80d0420 con 0x7fc6b8070ad0 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bfe8f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b80b1160 con 0x7fc6b8162230 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 0x7fc6b806c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 0x7fc6b806c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38062/0 (socket says 172.21.15.156:38062) 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 -- 172.21.15.156:0/2266727113 learned_addr learned my addr 172.21.15.156:0/2266727113 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6be42c640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b8162230 0x7fc6b80cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bdc2b640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b8070ad0 0x7fc6b8153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 -- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b8162230 msgr2=0x7fc6b80cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b8162230 0x7fc6b80cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 -- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b8070ad0 msgr2=0x7fc6b8153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b8070ad0 0x7fc6b8153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.830+0000 7fc6bd42a640 1 -- 172.21.15.156:0/2266727113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b80ce070 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.831+0000 7fc6bd42a640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 0x7fc6b806c8e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c3f0 tx=0x7fc6a800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.831+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a80052d0 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.831+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a8005490 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.831+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b80ce350 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.831+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8007260 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.831+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b80ca670 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.832+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b80ca870 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.832+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc6a8007400 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.833+0000 7fc6ae7fc640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc69c06c530 0x7fc69c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.833+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fc6a808a6d0 con 0x7fc6b815c4f0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.833+0000 7fc6bdc2b640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc69c06c530 0x7fc69c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:47.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.833+0000 7fc6bdc2b640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc69c06c530 0x7fc69c06e9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4036000 tx=0x7fc6b4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:47.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:47.837+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc6b80ca870 con 0x7fc6b815c4f0 2024-03-29T04:08:48.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.051+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6b8068610 con 0x7fc6b815c4f0 2024-03-29T04:08:48.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.051+0000 7fc6ae7fc640 1 -- 172.21.15.156:0/2266727113 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7fc6a80589f0 con 0x7fc6b815c4f0 2024-03-29T04:08:48.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:48.031 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-29T04:08:48.031 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 2 2024-03-29T04:08:48.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc69c06c530 msgr2=0x7fc69c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:48.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc69c06c530 0x7fc69c06e9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4036000 tx=0x7fc6b4008030 comp rx=0 tx=0).stop 2024-03-29T04:08:48.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 msgr2=0x7fc6b806c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:48.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 0x7fc6b806c8e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c3f0 tx=0x7fc6a800c7f0 comp rx=0 tx=0).stop 2024-03-29T04:08:48.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 shutdown_connections 2024-03-29T04:08:48.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc69c06c530 0x7fc69c06e9f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:48.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6b8162230 0x7fc6b80cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:48.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b815c4f0 0x7fc6b806c8e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:48.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 --2- 172.21.15.156:0/2266727113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b8070ad0 0x7fc6b8153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:48.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 >> 172.21.15.156:0/2266727113 conn(0x7fc6b80d2250 msgr2=0x7fc6b80af040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:48.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 shutdown_connections 2024-03-29T04:08:48.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:48.053+0000 7fc6bfe8f640 1 -- 172.21.15.156:0/2266727113 wait complete. 2024-03-29T04:08:50.045 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:08:50.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.178+0000 7fa70f37c640 1 Processor -- start 2024-03-29T04:08:50.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.178+0000 7fa70f37c640 1 -- start start 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70f37c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 0x7fa708142690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70f37c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7080b30f0 con 0x7fa70815c650 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70f37c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa70815d020 con 0x7fa7080ad740 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70f37c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa708142bd0 con 0x7fa70815c270 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70c917640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa70815c650 0x7fa7080ad030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48762/0 (socket says 172.21.15.156:48762) 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70c917640 1 -- 172.21.15.156:0/1302456516 learned_addr learned my addr 172.21.15.156:0/1302456516 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831216047 0 0) 0x7fa7080b30f0 con 0x7fa70815c650 2024-03-29T04:08:50.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f4003520 con 0x7fa70815c650 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa70d118640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 0x7fa708142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4027375945 0 0) 0x7fa6f4003520 con 0x7fa70815c650 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 msgr2=0x7fa708142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 0x7fa708142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 >> v1:172.21.15.203:6789/0 conn(0x7fa7080ad740 legacy=0x7fa7080b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70815d300 con 0x7fa70815c650 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fa6f8001670 con 0x7fa70815c650 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.179+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 >> v1:172.21.15.156:6789/0 conn(0x7fa70815c650 legacy=0x7fa7080ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa6fffff640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f40039e0 0x7fa6f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa6fffff640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 0x7fa6f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa6fffff640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4004d40 0x7fa6f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa708142bd0 con 0x7fa6f40039e0 2024-03-29T04:08:50.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f400bb50 con 0x7fa6f4004d40 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f400bd50 con 0x7fa70815c270 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d118640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f40039e0 0x7fa6f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70c917640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 0x7fa6f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d919640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4004d40 0x7fa6f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d118640 1 -- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 msgr2=0x7fa6f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d118640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815c270 0x7fa6f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d118640 1 -- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4004d40 msgr2=0x7fa6f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d118640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4004d40 0x7fa6f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.180+0000 7fa70d118640 1 -- 172.21.15.156:0/1302456516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f4003520 con 0x7fa6f40039e0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70d118640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f40039e0 0x7fa6f4003e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa704009ac0 tx=0x7fa6f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbd9195ce4e98d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704039070 con 0x7fa6f40039e0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa70402c6b0 con 0x7fa6f40039e0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa6fffff640 1 -- 172.21.15.156:0/1302456516 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704009690 con 0x7fa6f40039e0 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70f37c640 1 -- 172.21.15.156:0/1302456516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f40039e0 msgr2=0x7fa6f4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70f37c640 1 --2- 172.21.15.156:0/1302456516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f40039e0 0x7fa6f4003e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa704009ac0 tx=0x7fa6f4005830 comp rx=0 tx=0).stop 2024-03-29T04:08:50.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70d919640 1 -- 172.21.15.156:0/1302456516 reap_dead start 2024-03-29T04:08:50.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70f37c640 1 -- 172.21.15.156:0/1302456516 shutdown_connections 2024-03-29T04:08:50.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70f37c640 1 -- 172.21.15.156:0/1302456516 >> 172.21.15.156:0/1302456516 conn(0x7fa7080b1d50 msgr2=0x7fa7080b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:50.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70f37c640 1 -- 172.21.15.156:0/1302456516 shutdown_connections 2024-03-29T04:08:50.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.181+0000 7fa70f37c640 1 -- 172.21.15.156:0/1302456516 wait complete. 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 Processor -- start 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 -- start start 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f40039e0 0x7fa7081f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 0x7fa7080ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080cedf0 0x7fa708144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa70815cad0 con 0x7fa7081f2180 2024-03-29T04:08:50.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa70815d8c0 con 0x7fa6f40039e0 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70f37c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa70815cc50 con 0x7fa7080cedf0 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70c917640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 0x7fa7080ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70c917640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 0x7fa7080ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38074/0 (socket says 172.21.15.156:38074) 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.183+0000 7fa70c917640 1 -- 172.21.15.156:0/4093081941 learned_addr learned my addr 172.21.15.156:0/4093081941 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70d118640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f40039e0 0x7fa7081f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70d919640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080cedf0 0x7fa708144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70c917640 1 -- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080cedf0 msgr2=0x7fa708144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70c917640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080cedf0 0x7fa708144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70c917640 1 -- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f40039e0 msgr2=0x7fa7081f4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70c917640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f40039e0 0x7fa7081f4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70c917640 1 -- 172.21.15.156:0/4093081941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7081f54d0 con 0x7fa7081f2180 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70d919640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080cedf0 0x7fa708144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70c917640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 0x7fa7080ce8b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8001a90 tx=0x7fa6f80077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800f990 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f8018070 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800fd50 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7080cc490 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.184+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7080cca50 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.185+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa708068610 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.186+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa6f801d460 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.186+0000 7fa6fdffb640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.186+0000 7fa70d118640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.186+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa6f808b330 con 0x7fa7081f2180 2024-03-29T04:08:50.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.186+0000 7fa70d118640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa704036000 tx=0x7fa704030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.191+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa6f8059ca0 con 0x7fa7081f2180 2024-03-29T04:08:50.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.404+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7080cf1d0 con 0x7fa7081f2180 2024-03-29T04:08:50.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.404+0000 7fa6fdffb640 1 -- 172.21.15.156:0/4093081941 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3873 (secure 0 0 0) 0x7fa6f80595a0 con 0x7fa7081f2180 2024-03-29T04:08:50.384 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:50.384 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-29T04:08:50.384 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 3 2024-03-29T04:08:50.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 msgr2=0x7fa6ec06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa704036000 tx=0x7fa704030000 comp rx=0 tx=0).stop 2024-03-29T04:08:50.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 msgr2=0x7fa7080ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 0x7fa7080ce8b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8001a90 tx=0x7fa6f80077a0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 shutdown_connections 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080cedf0 0x7fa708144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7081f2180 0x7fa7080ce8b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 --2- 172.21.15.156:0/4093081941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f40039e0 0x7fa7081f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 >> 172.21.15.156:0/4093081941 conn(0x7fa7080b1d50 msgr2=0x7fa7080afc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.406+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 shutdown_connections 2024-03-29T04:08:50.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.407+0000 7fa70f37c640 1 -- 172.21.15.156:0/4093081941 wait complete. 2024-03-29T04:08:50.397 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:50.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.518+0000 7fb34599d640 1 Processor -- start 2024-03-29T04:08:50.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.518+0000 7fb34599d640 1 -- start start 2024-03-29T04:08:50.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34599d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb3400d2840 0x7fb3400d2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34599d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400b31d0 0x7fb3400d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34599d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3401621b0 0x7fb340162640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34599d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb34015f960 con 0x7fb3401621b0 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34599d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb3400d3160 con 0x7fb3400d2840 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34599d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb34015ed50 con 0x7fb3400b31d0 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3401621b0 0x7fb340162640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3401621b0 0x7fb340162640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38076/0 (socket says 172.21.15.156:38076) 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 -- 172.21.15.156:0/803700973 learned_addr learned my addr 172.21.15.156:0/803700973 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb33ffff640 1 --2- 172.21.15.156:0/803700973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400b31d0 0x7fb3400d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34499b640 1 --2- 172.21.15.156:0/803700973 >> v2:172.21.15.203:3300/0 conn(0x7fb3400d2840 0x7fb3400d2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 -- 172.21.15.156:0/803700973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400b31d0 msgr2=0x7fb3400d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 --2- 172.21.15.156:0/803700973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400b31d0 0x7fb3400d4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 -- 172.21.15.156:0/803700973 >> v2:172.21.15.203:3300/0 conn(0x7fb3400d2840 msgr2=0x7fb3400d2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 --2- 172.21.15.156:0/803700973 >> v2:172.21.15.203:3300/0 conn(0x7fb3400d2840 0x7fb3400d2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.519+0000 7fb34519c640 1 -- 172.21.15.156:0/803700973 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34015f030 con 0x7fb3401621b0 2024-03-29T04:08:50.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34519c640 1 --2- 172.21.15.156:0/803700973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb340162640 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb338009e50 tx=0x7fb33802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31a8ed39cc6ab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb33f7fe640 1 -- 172.21.15.156:0/803700973 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33802ca70 con 0x7fb3401621b0 2024-03-29T04:08:50.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb33f7fe640 1 -- 172.21.15.156:0/803700973 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb33802cc10 con 0x7fb3401621b0 2024-03-29T04:08:50.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb33f7fe640 1 -- 172.21.15.156:0/803700973 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3380343f0 con 0x7fb3401621b0 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 -- 172.21.15.156:0/803700973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 msgr2=0x7fb340162640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 --2- 172.21.15.156:0/803700973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb340162640 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb338009e50 tx=0x7fb33802bed0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 -- 172.21.15.156:0/803700973 shutdown_connections 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 --2- 172.21.15.156:0/803700973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb340162640 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 --2- 172.21.15.156:0/803700973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400b31d0 0x7fb3400d4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 --2- 172.21.15.156:0/803700973 >> v2:172.21.15.203:3300/0 conn(0x7fb3400d2840 0x7fb3400d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 -- 172.21.15.156:0/803700973 >> 172.21.15.156:0/803700973 conn(0x7fb34009d5a0 msgr2=0x7fb3400d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.520+0000 7fb34599d640 1 -- 172.21.15.156:0/803700973 shutdown_connections 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.521+0000 7fb34599d640 1 -- 172.21.15.156:0/803700973 wait complete. 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 Processor -- start 2024-03-29T04:08:50.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 -- start start 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3400b31d0 0x7fb34014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400d2840 0x7fb3401f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb3400cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3400d50d0 con 0x7fb3401621b0 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb34015f5f0 con 0x7fb3400b31d0 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.522+0000 7fb34599d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb34015e9c0 con 0x7fb3400d2840 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb3400cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb3400cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38088/0 (socket says 172.21.15.156:38088) 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 -- 172.21.15.156:0/2974932162 learned_addr learned my addr 172.21.15.156:0/2974932162 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34499b640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3400b31d0 0x7fb34014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb33ffff640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400d2840 0x7fb3401f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 -- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400d2840 msgr2=0x7fb3401f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400d2840 0x7fb3401f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 -- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3400b31d0 msgr2=0x7fb34014b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3400b31d0 0x7fb34014b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 -- 172.21.15.156:0/2974932162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3400ce1c0 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb33ffff640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400d2840 0x7fb3401f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34499b640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3400b31d0 0x7fb34014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34519c640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb3400cdc80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb338036040 tx=0x7fb338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb338044020 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb338040670 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb338009510 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3400ce490 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.523+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3400b4ba0 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.524+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb34004f8f0 con 0x7fb3401621b0 2024-03-29T04:08:50.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.525+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb338047020 con 0x7fb3401621b0 2024-03-29T04:08:50.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.525+0000 7fb33d7fa640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb32406c530 0x7fb32406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.525+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fb3380ade00 con 0x7fb3401621b0 2024-03-29T04:08:50.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.525+0000 7fb34499b640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb32406c530 0x7fb32406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.525+0000 7fb34499b640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb32406c530 0x7fb32406e9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb334006fd0 tx=0x7fb334008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.529+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb33804e030 con 0x7fb3401621b0 2024-03-29T04:08:50.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.706+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3400d2620 con 0x7fb3401621b0 2024-03-29T04:08:50.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.707+0000 7fb33d7fa640 1 -- 172.21.15.156:0/2974932162 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7fb33807c800 con 0x7fb3401621b0 2024-03-29T04:08:50.687 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:50.687 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:43.774034+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb32406c530 msgr2=0x7fb32406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb32406c530 0x7fb32406e9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb334006fd0 tx=0x7fb334008040 comp rx=0 tx=0).stop 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 msgr2=0x7fb3400cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb3400cdc80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb338036040 tx=0x7fb338004a40 comp rx=0 tx=0).stop 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 shutdown_connections 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb32406c530 0x7fb32406e9f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3401621b0 0x7fb3400cdc80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3400d2840 0x7fb3401f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 --2- 172.21.15.156:0/2974932162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3400b31d0 0x7fb34014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 >> 172.21.15.156:0/2974932162 conn(0x7fb34009d5a0 msgr2=0x7fb34015ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:50.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 shutdown_connections 2024-03-29T04:08:50.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.709+0000 7fb34599d640 1 -- 172.21.15.156:0/2974932162 wait complete. 2024-03-29T04:08:50.700 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:08:50.700 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:08:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.827+0000 7f56a9d3d640 1 Processor -- start 2024-03-29T04:08:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.827+0000 7f56a9d3d640 1 -- start start 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a9d3d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f56a40d2120 0x7f56a406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a9d3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a40cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a9d3d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40d2df0 con 0x7f56a415e920 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a9d3d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f56a40cf7a0 con 0x7f56a40d2120 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a9d3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f56a40cf190 con 0x7f56a406cb10 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a37fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f56a415e920 0x7f56a40d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48768/0 (socket says 172.21.15.156:48768) 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a37fe640 1 -- 172.21.15.156:0/3566565600 learned_addr learned my addr 172.21.15.156:0/3566565600 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f569bfff640 1 --2- 172.21.15.156:0/3566565600 >> v2:172.21.15.156:3300/0 conn(0x7f56a40d2120 0x7f56a406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a2ffd640 1 -- 172.21.15.156:0/3566565600 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213101469 0 0) 0x7f56a40d2df0 con 0x7f56a415e920 2024-03-29T04:08:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a2ffd640 1 -- 172.21.15.156:0/3566565600 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5690003520 con 0x7f56a415e920 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f56a3fff640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a40cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f569bfff640 1 -- 172.21.15.156:0/3566565600 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 msgr2=0x7f56a40cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.828+0000 7f569bfff640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a40cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f569bfff640 1 -- 172.21.15.156:0/3566565600 >> v1:172.21.15.156:6789/0 conn(0x7f56a415e920 legacy=0x7f56a40d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f569bfff640 1 -- 172.21.15.156:0/3566565600 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a40d3080 con 0x7f56a40d2120 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f56a3fff640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a40cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f569bfff640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a406c5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f568c0097a0 tx=0x7f568c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159442591acf1083 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f56a2ffd640 1 -- 172.21.15.156:0/3566565600 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c035020 con 0x7f56a40d2120 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f56a2ffd640 1 -- 172.21.15.156:0/3566565600 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c031830 con 0x7f56a40d2120 2024-03-29T04:08:50.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.829+0000 7f56a2ffd640 1 -- 172.21.15.156:0/3566565600 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c031c00 con 0x7f56a40d2120 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 -- 172.21.15.156:0/3566565600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 msgr2=0x7f56a406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a406c5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f568c0097a0 tx=0x7f568c02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 -- 172.21.15.156:0/3566565600 shutdown_connections 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a40cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/3566565600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a406c5d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 -- 172.21.15.156:0/3566565600 >> 172.21.15.156:0/3566565600 conn(0x7f56a409d7a0 msgr2=0x7f56a40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 -- 172.21.15.156:0/3566565600 shutdown_connections 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.830+0000 7f56a9d3d640 1 -- 172.21.15.156:0/3566565600 wait complete. 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.831+0000 7f56a9d3d640 1 Processor -- start 2024-03-29T04:08:50.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.831+0000 7f56a9d3d640 1 -- start start 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a9d3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a4152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a9d3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a40b7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a9d3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a415e920 0x7f56a40b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a9d3d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56a40d34e0 con 0x7f56a40d2120 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a9d3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56a40d3360 con 0x7f56a415e920 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a9d3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f56a40d0170 con 0x7f56a406cb10 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a40b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a40b7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38108/0 (socket says 172.21.15.156:38108) 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 -- 172.21.15.156:0/2781637105 learned_addr learned my addr 172.21.15.156:0/2781637105 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a37fe640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a4152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f56a3fff640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a415e920 0x7f56a40b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 -- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 msgr2=0x7f56a4152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a4152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 -- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a415e920 msgr2=0x7f56a40b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a415e920 0x7f56a40b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.832+0000 7f569bfff640 1 -- 172.21.15.156:0/2781637105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a4161b30 con 0x7f56a40d2120 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.833+0000 7f569bfff640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a40b7530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f568c02bee0 tx=0x7f568c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.833+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c035040 con 0x7f56a40d2120 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.833+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c03b920 con 0x7f56a40d2120 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.833+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a40b9830 con 0x7f56a40d2120 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.833+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c03bd10 con 0x7f56a40d2120 2024-03-29T04:08:50.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.833+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a41617c0 con 0x7f56a40d2120 2024-03-29T04:08:50.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.834+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56a40a2480 con 0x7f56a40d2120 2024-03-29T04:08:50.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.834+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f568c038020 con 0x7f56a40d2120 2024-03-29T04:08:50.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.835+0000 7f56a0ff9640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f568406c530 0x7f568406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:50.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.835+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f568c0ad4c0 con 0x7f56a40d2120 2024-03-29T04:08:50.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.835+0000 7f56a37fe640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f568406c530 0x7f568406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:50.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.835+0000 7f56a37fe640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f568406c530 0x7f568406e9f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5694007fd0 tx=0x7f5694009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:50.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:50.839+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f568c07be40 con 0x7f56a40d2120 2024-03-29T04:08:50.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.012+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f56a415d1f0 con 0x7f56a40d2120 2024-03-29T04:08:50.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.012+0000 7f56a0ff9640 1 -- 172.21.15.156:0/2781637105 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f568c031800 con 0x7f56a40d2120 2024-03-29T04:08:50.992 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:08:50.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f568406c530 msgr2=0x7f568406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f568406c530 0x7f568406e9f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5694007fd0 tx=0x7f5694009040 comp rx=0 tx=0).stop 2024-03-29T04:08:50.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 msgr2=0x7f56a40b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:50.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a40b7530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f568c02bee0 tx=0x7f568c0046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 shutdown_connections 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f568406c530 0x7f568406e9f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a415e920 0x7f56a40b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56a40d2120 0x7f56a40b7530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 --2- 172.21.15.156:0/2781637105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56a406cb10 0x7f56a4152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 >> 172.21.15.156:0/2781637105 conn(0x7f56a409d7a0 msgr2=0x7f56a40b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 shutdown_connections 2024-03-29T04:08:50.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.014+0000 7f56a9d3d640 1 -- 172.21.15.156:0/2781637105 wait complete. 2024-03-29T04:08:51.005 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T04:08:51.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.126+0000 7fd30a582640 1 Processor -- start 2024-03-29T04:08:51.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.126+0000 7fd30a582640 1 -- start start 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd30a582640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd3040700e0 0x7fd30409b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd30a582640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd30406fa20 0x7fd3040704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd30a582640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd3041622a0 0x7fd3041626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd30a582640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd30415ec60 con 0x7fd3041622a0 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd30a582640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd30415fc60 con 0x7fd3040700e0 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd30a582640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd304070a00 con 0x7fd30406fa20 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd3041622a0 0x7fd3041626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd3041622a0 0x7fd3041626f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38118/0 (socket says 172.21.15.156:38118) 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 -- 172.21.15.156:0/562685417 learned_addr learned my addr 172.21.15.156:0/562685417 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd303fff640 1 --2- 172.21.15.156:0/562685417 >> v2:172.21.15.203:3300/0 conn(0x7fd3040700e0 0x7fd30409b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd3037fe640 1 --2- 172.21.15.156:0/562685417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd30406fa20 0x7fd3040704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 -- 172.21.15.156:0/562685417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd30406fa20 msgr2=0x7fd3040704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 --2- 172.21.15.156:0/562685417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd30406fa20 0x7fd3040704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 -- 172.21.15.156:0/562685417 >> v2:172.21.15.203:3300/0 conn(0x7fd3040700e0 msgr2=0x7fd30409b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 --2- 172.21.15.156:0/562685417 >> v2:172.21.15.203:3300/0 conn(0x7fd3040700e0 0x7fd30409b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.127+0000 7fd308b1f640 1 -- 172.21.15.156:0/562685417 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3040b19e0 con 0x7fd3041622a0 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.128+0000 7fd308b1f640 1 --2- 172.21.15.156:0/562685417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3041626f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd304073070 tx=0x7fd2fc02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d12836600d5290 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.128+0000 7fd302ffd640 1 -- 172.21.15.156:0/562685417 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc02c950 con 0x7fd3041622a0 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.128+0000 7fd302ffd640 1 -- 172.21.15.156:0/562685417 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc02caf0 con 0x7fd3041622a0 2024-03-29T04:08:51.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.128+0000 7fd302ffd640 1 -- 172.21.15.156:0/562685417 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc0343f0 con 0x7fd3041622a0 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 -- 172.21.15.156:0/562685417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 msgr2=0x7fd3041626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 --2- 172.21.15.156:0/562685417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3041626f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd304073070 tx=0x7fd2fc02bdb0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 -- 172.21.15.156:0/562685417 shutdown_connections 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 --2- 172.21.15.156:0/562685417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3041626f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 --2- 172.21.15.156:0/562685417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd30406fa20 0x7fd3040704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 --2- 172.21.15.156:0/562685417 >> v2:172.21.15.203:3300/0 conn(0x7fd3040700e0 0x7fd30409b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 -- 172.21.15.156:0/562685417 >> 172.21.15.156:0/562685417 conn(0x7fd3040b3d30 msgr2=0x7fd3040b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 -- 172.21.15.156:0/562685417 shutdown_connections 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.129+0000 7fd30a582640 1 -- 172.21.15.156:0/562685417 wait complete. 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.130+0000 7fd30a582640 1 Processor -- start 2024-03-29T04:08:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.130+0000 7fd30a582640 1 -- start start 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd30a582640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd30406fa20 0x7fd3041505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd30a582640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3040700e0 0x7fd3040b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd30a582640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3040b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd30a582640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd30415f460 con 0x7fd3041622a0 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd30a582640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd30415f180 con 0x7fd30406fa20 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd30a582640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd30415f740 con 0x7fd3040700e0 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3040b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3040b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38120/0 (socket says 172.21.15.156:38120) 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 -- 172.21.15.156:0/754022838 learned_addr learned my addr 172.21.15.156:0/754022838 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd3037fe640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3040700e0 0x7fd3040b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd303fff640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd30406fa20 0x7fd3041505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 -- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3040700e0 msgr2=0x7fd3040b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3040700e0 0x7fd3040b6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 -- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd30406fa20 msgr2=0x7fd3041505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd30406fa20 0x7fd3041505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 -- 172.21.15.156:0/754022838 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3040b75f0 con 0x7fd3041622a0 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd303fff640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd30406fa20 0x7fd3041505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.131+0000 7fd308b1f640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3040b2cf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009f40 tx=0x7fd2fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.132+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc043020 con 0x7fd3041622a0 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.132+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc03e450 con 0x7fd3041622a0 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.132+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc03e840 con 0x7fd3041622a0 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.132+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3040b78d0 con 0x7fd3041622a0 2024-03-29T04:08:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.132+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3040b7130 con 0x7fd3041622a0 2024-03-29T04:08:51.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.133+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd304068610 con 0x7fd3041622a0 2024-03-29T04:08:51.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.133+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd2fc046020 con 0x7fd3041622a0 2024-03-29T04:08:51.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.133+0000 7fd300ff9640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd2f006c560 0x7fd2f006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.134+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd2fc0ada60 con 0x7fd3041622a0 2024-03-29T04:08:51.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.134+0000 7fd303fff640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd2f006c560 0x7fd2f006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.134+0000 7fd303fff640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd2f006c560 0x7fd2f006ea20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80099a0 tx=0x7fd2f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.138+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd2fc04d030 con 0x7fd3041622a0 2024-03-29T04:08:51.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.313+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd3040c8e10 con 0x7fd3041622a0 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.313+0000 7fd300ff9640 1 -- 172.21.15.156:0/754022838 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fd2fc07c3e0 con 0x7fd3041622a0 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stdout:[ 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stdout: "poolnum": 1, 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stdout: "poolname": ".mgr" 2024-03-29T04:08:51.293 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:08:51.294 INFO:teuthology.orchestra.run.smithi156.stdout:] 2024-03-29T04:08:51.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd2f006c560 msgr2=0x7fd2f006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd2f006c560 0x7fd2f006ea20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80099a0 tx=0x7fd2f8008040 comp rx=0 tx=0).stop 2024-03-29T04:08:51.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 msgr2=0x7fd3040b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3040b2cf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009f40 tx=0x7fd2fc004a40 comp rx=0 tx=0).stop 2024-03-29T04:08:51.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 shutdown_connections 2024-03-29T04:08:51.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd2f006c560 0x7fd2f006ea20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3041622a0 0x7fd3040b2cf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3040700e0 0x7fd3040b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 --2- 172.21.15.156:0/754022838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd30406fa20 0x7fd3041505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 >> 172.21.15.156:0/754022838 conn(0x7fd3040b3d30 msgr2=0x7fd30406cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:51.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 shutdown_connections 2024-03-29T04:08:51.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.315+0000 7fd30a582640 1 -- 172.21.15.156:0/754022838 wait complete. 2024-03-29T04:08:51.306 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T04:08:51.306 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T04:08:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.431+0000 7fc023ec5640 1 Processor -- start 2024-03-29T04:08:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.431+0000 7fc023ec5640 1 -- start start 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc023ec5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c0d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc023ec5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc01c09b650 0x7fc01c0b3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc023ec5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01c0b3bd0 con 0x7fc01c09ae20 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc023ec5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc01c0b4d90 con 0x7fc01c09b650 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc023ec5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c15f780 con 0x7fc01c09aa40 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021460640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc01c09b650 0x7fc01c0b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc022462640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc01c09ae20 0x7fc01c0b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48782/0 (socket says 172.21.15.156:48782) 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc022462640 1 -- 172.21.15.156:0/707844637 learned_addr learned my addr 172.21.15.156:0/707844637 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021c61640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c0d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021460640 1 -- 172.21.15.156:0/707844637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 msgr2=0x7fc01c0d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:51.413 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021460640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c0d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.413 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021460640 1 -- 172.21.15.156:0/707844637 >> v1:172.21.15.156:6789/0 conn(0x7fc01c09ae20 legacy=0x7fc01c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.413 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021460640 1 -- 172.21.15.156:0/707844637 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c15fa60 con 0x7fc01c09b650 2024-03-29T04:08:51.413 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc020c5f640 1 -- 172.21.15.156:0/707844637 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903612066 0 0) 0x7fc01c0b3bd0 con 0x7fc01c09ae20 2024-03-29T04:08:51.413 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.432+0000 7fc021c61640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c0d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:51.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.433+0000 7fc021460640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c0b3690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0097a0 tx=0x7fc00c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a058f439e0308e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.433+0000 7fc020c5f640 1 -- 172.21.15.156:0/707844637 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c039070 con 0x7fc01c09b650 2024-03-29T04:08:51.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.433+0000 7fc020c5f640 1 -- 172.21.15.156:0/707844637 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c0316f0 con 0x7fc01c09b650 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.433+0000 7fc020c5f640 1 -- 172.21.15.156:0/707844637 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c031ac0 con 0x7fc01c09b650 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 -- 172.21.15.156:0/707844637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 msgr2=0x7fc01c0b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c0b3690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0097a0 tx=0x7fc00c02b790 comp rx=0 tx=0).stop 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 -- 172.21.15.156:0/707844637 shutdown_connections 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c0b3690 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 --2- 172.21.15.156:0/707844637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c0d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 -- 172.21.15.156:0/707844637 >> 172.21.15.156:0/707844637 conn(0x7fc01c0b3ed0 msgr2=0x7fc01c0b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 -- 172.21.15.156:0/707844637 shutdown_connections 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.434+0000 7fc023ec5640 1 -- 172.21.15.156:0/707844637 wait complete. 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.435+0000 7fc023ec5640 1 Processor -- start 2024-03-29T04:08:51.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.435+0000 7fc023ec5640 1 -- start start 2024-03-29T04:08:51.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.435+0000 7fc023ec5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c14dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc01c09ae20 0x7fc01c0b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c15b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c09b2b0 con 0x7fc01c09b650 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c15ea30 con 0x7fc01c09ae20 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c0b49f0 con 0x7fc01c09aa40 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c15b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38140/0 (socket says 172.21.15.156:38140) 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 -- 172.21.15.156:0/486157513 learned_addr learned my addr 172.21.15.156:0/486157513 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc021c61640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c14dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc021460640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc01c09ae20 0x7fc01c0b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 -- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 msgr2=0x7fc01c14dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c14dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 -- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc01c09ae20 msgr2=0x7fc01c0b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc01c09ae20 0x7fc01c0b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 -- 172.21.15.156:0/486157513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c152ed0 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc022462640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c15b480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc01400c7a0 tx=0x7fc01400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014018070 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc01400a640 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01c1f4ab0 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01400aa70 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.436+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c0b4fa0 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.437+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc01c068610 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.438+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc01401b070 con 0x7fc01c09b650 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.438+0000 7fc0127fc640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc00006c530 0x7fc00006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.438+0000 7fc021c61640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc00006c530 0x7fc00006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.438+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fc01408a8a0 con 0x7fc01c09b650 2024-03-29T04:08:51.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.439+0000 7fc021c61640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc00006c530 0x7fc00006e9f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc018006fd0 tx=0x7fc018008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.422 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.442+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc014058bc0 con 0x7fc01c09b650 2024-03-29T04:08:51.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.617+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc01c1600d0 con 0x7fc01c09b650 2024-03-29T04:08:51.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.787+0000 7fc0127fc640 1 -- 172.21.15.156:0/486157513 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fc01405cec0 con 0x7fc01c09b650 2024-03-29T04:08:51.767 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_metadata' created 2024-03-29T04:08:51.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc00006c530 msgr2=0x7fc00006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc00006c530 0x7fc00006e9f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc018006fd0 tx=0x7fc018008040 comp rx=0 tx=0).stop 2024-03-29T04:08:51.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 msgr2=0x7fc01c15b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c15b480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc01400c7a0 tx=0x7fc01400cc60 comp rx=0 tx=0).stop 2024-03-29T04:08:51.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 shutdown_connections 2024-03-29T04:08:51.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc00006c530 0x7fc00006e9f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c09b650 0x7fc01c15b480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc01c09ae20 0x7fc01c0b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 --2- 172.21.15.156:0/486157513 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc01c09aa40 0x7fc01c14dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 >> 172.21.15.156:0/486157513 conn(0x7fc01c0b3ed0 msgr2=0x7fc01c15ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:51.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 shutdown_connections 2024-03-29T04:08:51.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.790+0000 7fc023ec5640 1 -- 172.21.15.156:0/486157513 wait complete. 2024-03-29T04:08:51.784 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T04:08:51.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.913+0000 7fb463f5f640 1 Processor -- start 2024-03-29T04:08:51.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb463f5f640 1 -- start start 2024-03-29T04:08:51.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb463f5f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c145c70 0x7fb45c0b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb463f5f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb45c0b4310 0x7fb45c0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb463f5f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb45c15cb10 con 0x7fb45c0cf340 2024-03-29T04:08:51.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb463f5f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb45c0b4770 con 0x7fb45c0b4310 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb463f5f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb45c15cdf0 con 0x7fb45c145c70 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.914+0000 7fb4614fa640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb45c0b4310 0x7fb45c0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb4624fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb45c0cf340 0x7fb45c06f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58912/0 (socket says 172.21.15.156:58912) 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb4624fc640 1 -- 172.21.15.156:0/1271295448 learned_addr learned my addr 172.21.15.156:0/1271295448 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb461cfb640 1 --2- 172.21.15.156:0/1271295448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c145c70 0x7fb45c0b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb4614fa640 1 -- 172.21.15.156:0/1271295448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c145c70 msgr2=0x7fb45c0b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb4614fa640 1 --2- 172.21.15.156:0/1271295448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c145c70 0x7fb45c0b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb4614fa640 1 -- 172.21.15.156:0/1271295448 >> v1:172.21.15.156:6789/0 conn(0x7fb45c0cf340 legacy=0x7fb45c06f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb4614fa640 1 -- 172.21.15.156:0/1271295448 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c15d0d0 con 0x7fb45c0b4310 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.915+0000 7fb460cf9640 1 -- 172.21.15.156:0/1271295448 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603085194 0 0) 0x7fb45c15cb10 con 0x7fb45c0cf340 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb4614fa640 1 --2- 172.21.15.156:0/1271295448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0cee00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0097a0 tx=0x7fb44c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15216c90b637ec3e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb460cf9640 1 -- 172.21.15.156:0/1271295448 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c0315a0 con 0x7fb45c0b4310 2024-03-29T04:08:51.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb460cf9640 1 -- 172.21.15.156:0/1271295448 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb44c037070 con 0x7fb45c0b4310 2024-03-29T04:08:51.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb460cf9640 1 -- 172.21.15.156:0/1271295448 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c031960 con 0x7fb45c0b4310 2024-03-29T04:08:51.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 -- 172.21.15.156:0/1271295448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 msgr2=0x7fb45c0cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 --2- 172.21.15.156:0/1271295448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0cee00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0097a0 tx=0x7fb44c02b840 comp rx=0 tx=0).stop 2024-03-29T04:08:51.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 -- 172.21.15.156:0/1271295448 shutdown_connections 2024-03-29T04:08:51.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 --2- 172.21.15.156:0/1271295448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0cee00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 --2- 172.21.15.156:0/1271295448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c145c70 0x7fb45c0b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 -- 172.21.15.156:0/1271295448 >> 172.21.15.156:0/1271295448 conn(0x7fb45c0b7660 msgr2=0x7fb45c0b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 -- 172.21.15.156:0/1271295448 shutdown_connections 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.916+0000 7fb463f5f640 1 -- 172.21.15.156:0/1271295448 wait complete. 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.917+0000 7fb463f5f640 1 Processor -- start 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.917+0000 7fb463f5f640 1 -- start start 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb463f5f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb463f5f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c0cf340 0x7fb45c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb463f5f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c145c70 0x7fb45c06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb463f5f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0cf930 con 0x7fb45c0b4310 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb463f5f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0cf7b0 con 0x7fb45c145c70 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb463f5f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb45c146050 con 0x7fb45c0cf340 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb461cfb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb461cfb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45150/0 (socket says 172.21.15.156:45150) 2024-03-29T04:08:51.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb461cfb640 1 -- 172.21.15.156:0/2222787070 learned_addr learned my addr 172.21.15.156:0/2222787070 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb4614fa640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c0cf340 0x7fb45c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb4624fc640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c145c70 0x7fb45c06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb461cfb640 1 -- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c0cf340 msgr2=0x7fb45c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb461cfb640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c0cf340 0x7fb45c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.918+0000 7fb461cfb640 1 -- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c145c70 msgr2=0x7fb45c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb461cfb640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c145c70 0x7fb45c06c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb461cfb640 1 -- 172.21.15.156:0/2222787070 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c0ce040 con 0x7fb45c0b4310 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb4624fc640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c145c70 0x7fb45c06c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:51.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb461cfb640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0b2b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb458007f20 tx=0x7fb45800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458014620 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb458019070 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c142f40 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458014a00 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.919+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c0cda80 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.920+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45c04f9d0 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.921+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb458014ba0 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.921+0000 7fb4527fc640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb44006c530 0x7fb44006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.921+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fb45808bdc0 con 0x7fb45c0b4310 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.921+0000 7fb4614fa640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb44006c530 0x7fb44006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:51.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.921+0000 7fb4614fa640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb44006c530 0x7fb44006e9f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb44c036000 tx=0x7fb44c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:51.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:51.926+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb45805a650 con 0x7fb45c0b4310 2024-03-29T04:08:52.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.100+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb45c0d2f60 con 0x7fb45c0b4310 2024-03-29T04:08:52.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.790+0000 7fb4527fc640 1 -- 172.21.15.156:0/2222787070 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fb458059f50 con 0x7fb45c0b4310 2024-03-29T04:08:52.769 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_data' created 2024-03-29T04:08:52.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb44006c530 msgr2=0x7fb44006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:52.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb44006c530 0x7fb44006e9f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb44c036000 tx=0x7fb44c03d040 comp rx=0 tx=0).stop 2024-03-29T04:08:52.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 msgr2=0x7fb45c0b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:52.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0b2b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb458007f20 tx=0x7fb45800e570 comp rx=0 tx=0).stop 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 shutdown_connections 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb44006c530 0x7fb44006e9f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c145c70 0x7fb45c06c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb45c0cf340 0x7fb45c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 --2- 172.21.15.156:0/2222787070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c0b4310 0x7fb45c0b2b80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.792+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 >> 172.21.15.156:0/2222787070 conn(0x7fb45c0b7660 msgr2=0x7fb45c0b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.793+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 shutdown_connections 2024-03-29T04:08:52.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.793+0000 7fb463f5f640 1 -- 172.21.15.156:0/2222787070 wait complete. 2024-03-29T04:08:52.800 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T04:08:52.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.925+0000 7ff21ecf2640 1 Processor -- start 2024-03-29T04:08:52.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.925+0000 7ff21ecf2640 1 -- start start 2024-03-29T04:08:52.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ecf2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff218070480 0x7ff21806f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:52.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ecf2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff21806b5b0 0x7ff21806ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:52.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ecf2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2180b5ed0 con 0x7ff21815e960 2024-03-29T04:08:52.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ecf2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff21809aa40 con 0x7ff21806b5b0 2024-03-29T04:08:52.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ecf2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff21809ac90 con 0x7ff218070480 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ca8e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff21815e960 0x7ff2180b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58916/0 (socket says 172.21.15.156:58916) 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21ca8e640 1 -- 172.21.15.156:0/1909799644 learned_addr learned my addr 172.21.15.156:0/1909799644 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21d28f640 1 --2- 172.21.15.156:0/1909799644 >> v2:172.21.15.203:3300/0 conn(0x7ff21806b5b0 0x7ff21806ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff217fff640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff218070480 0x7ff21806f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff2177fe640 1 -- 172.21.15.156:0/1909799644 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002614423 0 0) 0x7ff2180b5ed0 con 0x7ff21815e960 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff2177fe640 1 -- 172.21.15.156:0/1909799644 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff208003520 con 0x7ff21815e960 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21d28f640 1 -- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff218070480 msgr2=0x7ff21806f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21d28f640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff218070480 0x7ff21806f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21d28f640 1 -- 172.21.15.156:0/1909799644 >> v1:172.21.15.156:6789/0 conn(0x7ff21815e960 legacy=0x7ff2180b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff21d28f640 1 -- 172.21.15.156:0/1909799644 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2180b6170 con 0x7ff21806b5b0 2024-03-29T04:08:52.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.926+0000 7ff217fff640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff218070480 0x7ff21806f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.927+0000 7ff21d28f640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff21806b5b0 0x7ff21806ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff210009d90 tx=0x7ff21002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15894b45fb34ebfa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.927+0000 7ff2177fe640 1 -- 172.21.15.156:0/1909799644 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210039070 con 0x7ff21806b5b0 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.927+0000 7ff2177fe640 1 -- 172.21.15.156:0/1909799644 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff21002cc50 con 0x7ff21806b5b0 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.927+0000 7ff2177fe640 1 -- 172.21.15.156:0/1909799644 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210034440 con 0x7ff21806b5b0 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 -- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff21806b5b0 msgr2=0x7ff21806ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff21806b5b0 0x7ff21806ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff210009d90 tx=0x7ff21002bea0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 -- 172.21.15.156:0/1909799644 shutdown_connections 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff21806b5b0 0x7ff21806ba40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/1909799644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff218070480 0x7ff21806f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 -- 172.21.15.156:0/1909799644 >> 172.21.15.156:0/1909799644 conn(0x7ff218072ce0 msgr2=0x7ff2180b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 -- 172.21.15.156:0/1909799644 shutdown_connections 2024-03-29T04:08:52.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.928+0000 7ff21ecf2640 1 -- 172.21.15.156:0/1909799644 wait complete. 2024-03-29T04:08:52.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.929+0000 7ff21ecf2640 1 Processor -- start 2024-03-29T04:08:52.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.929+0000 7ff21ecf2640 1 -- start start 2024-03-29T04:08:52.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ecf2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff21806b5b0 0x7ff218158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ecf2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218070480 0x7ff2181618f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ecf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 0x7ff218143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ecf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff2180b49e0 con 0x7ff21815e960 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ecf2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2180701f0 con 0x7ff218070480 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ecf2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff21815ed40 con 0x7ff21806b5b0 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 0x7ff218143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 0x7ff218143410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45162/0 (socket says 172.21.15.156:45162) 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 -- 172.21.15.156:0/2746824396 learned_addr learned my addr 172.21.15.156:0/2746824396 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff217fff640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218070480 0x7ff2181618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21ca8e640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff21806b5b0 0x7ff218158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 -- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff21806b5b0 msgr2=0x7ff218158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff21806b5b0 0x7ff218158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 -- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218070480 msgr2=0x7ff2181618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:52.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218070480 0x7ff2181618f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.930+0000 7ff21d28f640 1 -- 172.21.15.156:0/2746824396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff218143950 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.931+0000 7ff21d28f640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 0x7ff218143410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff210009d90 tx=0x7ff210004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.931+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210039050 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.931+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff210045070 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.931+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff218141b00 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.931+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210009450 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.931+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff218158ab0 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.932+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff21809f3c0 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.932+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff21003e610 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.933+0000 7ff2157fa640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff1fc06c530 0x7ff1fc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.933+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7ff2100adf50 con 0x7ff21815e960 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.933+0000 7ff21ca8e640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff1fc06c530 0x7ff1fc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:52.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.933+0000 7ff21ca8e640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff1fc06c530 0x7ff1fc06e9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff2180d0610 tx=0x7ff20c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:52.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:52.937+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff21007c660 con 0x7ff21815e960 2024-03-29T04:08:53.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.150+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff21804f9d0 con 0x7ff21815e960 2024-03-29T04:08:53.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.808+0000 7ff2157fa640 1 -- 172.21.15.156:0/2746824396 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7ff21007bf60 con 0x7ff21815e960 2024-03-29T04:08:53.788 INFO:teuthology.orchestra.run.smithi156.stderr:new fs with metadata pool 2 and data pool 3 2024-03-29T04:08:53.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.820+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff1fc06c530 msgr2=0x7ff1fc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:53.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.820+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff1fc06c530 0x7ff1fc06e9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff2180d0610 tx=0x7ff20c009040 comp rx=0 tx=0).stop 2024-03-29T04:08:53.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.820+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 msgr2=0x7ff218143410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:53.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.820+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 0x7ff218143410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff210009d90 tx=0x7ff210004a40 comp rx=0 tx=0).stop 2024-03-29T04:08:53.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.823+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 shutdown_connections 2024-03-29T04:08:53.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.823+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff1fc06c530 0x7ff1fc06e9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.823+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21815e960 0x7ff218143410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.823+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218070480 0x7ff2181618f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.823+0000 7ff21ecf2640 1 --2- 172.21.15.156:0/2746824396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff21806b5b0 0x7ff218158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.823+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 >> 172.21.15.156:0/2746824396 conn(0x7ff218072ce0 msgr2=0x7ff2181599e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:53.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.826+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 shutdown_connections 2024-03-29T04:08:53.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.826+0000 7ff21ecf2640 1 -- 172.21.15.156:0/2746824396 wait complete. 2024-03-29T04:08:53.820 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:53.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.942+0000 7ff3a2fba640 1 Processor -- start 2024-03-29T04:08:53.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.942+0000 7ff3a2fba640 1 -- start start 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a2fba640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff39c15c5f0 0x7ff39c0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a2fba640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff39c0cdf40 0x7ff39c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a2fba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c09aed0 con 0x7ff39c09aa40 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a2fba640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff39c09bae0 con 0x7ff39c15c5f0 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a2fba640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff39c09b1b0 con 0x7ff39c0cdf40 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff39c15c5f0 0x7ff39c0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff39c15c5f0 0x7ff39c0cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45170/0 (socket says 172.21.15.156:45170) 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 -- 172.21.15.156:0/2024897353 learned_addr learned my addr 172.21.15.156:0/2024897353 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a1557640 1 --2- 172.21.15.156:0/2024897353 >> v2:172.21.15.203:3300/0 conn(0x7ff39c0cdf40 0x7ff39c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 -- 172.21.15.156:0/2024897353 >> v2:172.21.15.203:3300/0 conn(0x7ff39c0cdf40 msgr2=0x7ff39c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 --2- 172.21.15.156:0/2024897353 >> v2:172.21.15.203:3300/0 conn(0x7ff39c0cdf40 0x7ff39c15d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 -- 172.21.15.156:0/2024897353 >> v1:172.21.15.203:6789/0 conn(0x7ff39c09aa40 legacy=0x7ff39c15c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:53.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a0d56640 1 -- 172.21.15.156:0/2024897353 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c09b490 con 0x7ff39c15c5f0 2024-03-29T04:08:53.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.943+0000 7ff3a1557640 1 --2- 172.21.15.156:0/2024897353 >> v2:172.21.15.203:3300/0 conn(0x7ff39c0cdf40 0x7ff39c15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:53.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.944+0000 7ff3a0d56640 1 --2- 172.21.15.156:0/2024897353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c15c5f0 0x7ff39c0cda00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0cc830 tx=0x7ff39002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee467ab4f131b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:53.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.944+0000 7ff39b7fe640 1 -- 172.21.15.156:0/2024897353 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390035020 con 0x7ff39c15c5f0 2024-03-29T04:08:53.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.944+0000 7ff39b7fe640 1 -- 172.21.15.156:0/2024897353 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff390031870 con 0x7ff39c15c5f0 2024-03-29T04:08:53.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.944+0000 7ff39b7fe640 1 -- 172.21.15.156:0/2024897353 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390031c40 con 0x7ff39c15c5f0 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 -- 172.21.15.156:0/2024897353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c15c5f0 msgr2=0x7ff39c0cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/2024897353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c15c5f0 0x7ff39c0cda00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0cc830 tx=0x7ff39002b910 comp rx=0 tx=0).stop 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 -- 172.21.15.156:0/2024897353 shutdown_connections 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/2024897353 >> v2:172.21.15.203:3300/0 conn(0x7ff39c0cdf40 0x7ff39c15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/2024897353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c15c5f0 0x7ff39c0cda00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 -- 172.21.15.156:0/2024897353 >> 172.21.15.156:0/2024897353 conn(0x7ff39c0d2250 msgr2=0x7ff39c0d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 -- 172.21.15.156:0/2024897353 shutdown_connections 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.945+0000 7ff3a2fba640 1 -- 172.21.15.156:0/2024897353 wait complete. 2024-03-29T04:08:53.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.946+0000 7ff3a2fba640 1 Processor -- start 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.946+0000 7ff3a2fba640 1 -- start start 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a2fba640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff39c15c5f0 0x7ff39c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a2fba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff39c06e040 0x7ff39c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a2fba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 0x7ff39c0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a2fba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c0d0030 con 0x7ff39c06e4c0 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a2fba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c0cfeb0 con 0x7ff39c06e040 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a2fba640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff39c0d01b0 con 0x7ff39c15c5f0 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 0x7ff39c0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 0x7ff39c0b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45182/0 (socket says 172.21.15.156:45182) 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 -- 172.21.15.156:0/530343471 learned_addr learned my addr 172.21.15.156:0/530343471 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a0d56640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff39c15c5f0 0x7ff39c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff39bfff640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff39c06e040 0x7ff39c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 -- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff39c15c5f0 msgr2=0x7ff39c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:53.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff39c15c5f0 0x7ff39c158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 -- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff39c06e040 msgr2=0x7ff39c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff39c06e040 0x7ff39c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 -- 172.21.15.156:0/530343471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c0b52b0 con 0x7ff39c06e4c0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.947+0000 7ff3a1557640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 0x7ff39c0b0760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff39400ce80 tx=0x7ff39400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.948+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39401a070 con 0x7ff39c06e4c0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.948+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff39400d880 con 0x7ff39c06e4c0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.948+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39400dc70 con 0x7ff39c06e4c0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.948+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39c0b5590 con 0x7ff39c06e4c0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.948+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff39c0b5b50 con 0x7ff39c06e4c0 2024-03-29T04:08:53.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.949+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39c068610 con 0x7ff39c06e4c0 2024-03-29T04:08:53.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.949+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff39401d070 con 0x7ff39c06e4c0 2024-03-29T04:08:53.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.949+0000 7ff3997fa640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff38006c530 0x7ff38006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:53.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.949+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff39408bde0 con 0x7ff39c06e4c0 2024-03-29T04:08:53.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.949+0000 7ff3a0d56640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff38006c530 0x7ff38006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:53.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.950+0000 7ff3a0d56640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff38006c530 0x7ff38006e9f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff390036000 tx=0x7ff39003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:53.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:53.954+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff394059e40 con 0x7ff39c06e4c0 2024-03-29T04:08:54.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.129+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff39c0d3160 con 0x7ff39c06e4c0 2024-03-29T04:08:54.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.131+0000 7ff3997fa640 1 -- 172.21.15.156:0/530343471 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16642 (secure 0 0 0) 0x7ff39405e140 con 0x7ff39c06e4c0 2024-03-29T04:08:54.110 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:54.111 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":20,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:53.151435+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff38006c530 msgr2=0x7ff38006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff38006c530 0x7ff38006e9f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff390036000 tx=0x7ff39003c040 comp rx=0 tx=0).stop 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 msgr2=0x7ff39c0b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 0x7ff39c0b0760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff39400ce80 tx=0x7ff39400a710 comp rx=0 tx=0).stop 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 shutdown_connections 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff38006c530 0x7ff38006e9f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff39c06e4c0 0x7ff39c0b0760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff39c06e040 0x7ff39c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 --2- 172.21.15.156:0/530343471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff39c15c5f0 0x7ff39c158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 >> 172.21.15.156:0/530343471 conn(0x7ff39c0d2250 msgr2=0x7ff39c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:54.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 shutdown_connections 2024-03-29T04:08:54.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.133+0000 7ff3a2fba640 1 -- 172.21.15.156:0/530343471 wait complete. 2024-03-29T04:08:54.123 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:54.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.246+0000 7f7c588a5640 1 Processor -- start 2024-03-29T04:08:54.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.246+0000 7f7c588a5640 1 -- start start 2024-03-29T04:08:54.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c588a5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7c5406ba40 0x7f7c540b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c588a5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c541456c0 con 0x7f7c5406b5b0 2024-03-29T04:08:54.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c588a5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c540b03f0 con 0x7f7c541452e0 2024-03-29T04:08:54.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c588a5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7c540b06d0 con 0x7f7c5406ba40 2024-03-29T04:08:54.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c4bfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7c5406b5b0 0x7f7c540b1490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58928/0 (socket says 172.21.15.156:58928) 2024-03-29T04:08:54.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c4bfff640 1 -- 172.21.15.156:0/2400916694 learned_addr learned my addr 172.21.15.156:0/2400916694 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c52d9d640 1 --2- 172.21.15.156:0/2400916694 >> v2:172.21.15.203:3300/0 conn(0x7f7c5406ba40 0x7f7c540b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101001543 0 0) 0x7f7c541456c0 con 0x7f7c5406b5b0 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c40003520 con 0x7f7c5406b5b0 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 783665543 0 0) 0x7f7c40003520 con 0x7f7c5406b5b0 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 >> v2:172.21.15.203:3300/0 conn(0x7f7c5406ba40 msgr2=0x7f7c540b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 --2- 172.21.15.156:0/2400916694 >> v2:172.21.15.203:3300/0 conn(0x7f7c5406ba40 0x7f7c540b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 >> v1:172.21.15.203:6789/0 conn(0x7f7c541452e0 legacy=0x7f7c540af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c540b09b0 con 0x7f7c5406b5b0 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f7c3c001670 con 0x7f7c5406b5b0 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c52d9d640 1 --2- 172.21.15.156:0/2400916694 >> v2:172.21.15.203:3300/0 conn(0x7f7c5406ba40 0x7f7c540b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 >> v1:172.21.15.156:6789/0 conn(0x7f7c5406b5b0 legacy=0x7f7c540b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c400039b0 0x7f7c40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c400051c0 0x7f7c4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c540b06d0 con 0x7f7c40004800 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4000c060 con 0x7f7c400039b0 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.247+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4000c260 con 0x7f7c400051c0 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c52d9d640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c400051c0 0x7f7c4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c5259c640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c400039b0 0x7f7c40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 -- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c400051c0 msgr2=0x7f7c4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c400051c0 0x7f7c4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 -- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c400039b0 msgr2=0x7f7c40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c400039b0 0x7f7c40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 -- 172.21.15.156:0/2400916694 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c40003520 con 0x7f7c40004800 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c52d9d640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c400051c0 0x7f7c4000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c5259c640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c400039b0 0x7f7c40003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c52d9d640 1 -- 172.21.15.156:0/2400916694 reap_dead start 2024-03-29T04:08:54.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.248+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c40004c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00c210 tx=0x7f7c40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb6f3bec19ea74c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c007d30 con 0x7f7c40004800 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c030070 con 0x7f7c40004800 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c51d9b640 1 -- 172.21.15.156:0/2400916694 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c005d70 con 0x7f7c40004800 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 -- 172.21.15.156:0/2400916694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 msgr2=0x7f7c40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c40004c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00c210 tx=0x7f7c40005830 comp rx=0 tx=0).stop 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 -- 172.21.15.156:0/2400916694 shutdown_connections 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 --2- 172.21.15.156:0/2400916694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c40004c80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 -- 172.21.15.156:0/2400916694 >> 172.21.15.156:0/2400916694 conn(0x7f7c5409aa40 msgr2=0x7f7c540b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 -- 172.21.15.156:0/2400916694 shutdown_connections 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.249+0000 7f7c588a5640 1 -- 172.21.15.156:0/2400916694 wait complete. 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.250+0000 7f7c588a5640 1 Processor -- start 2024-03-29T04:08:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 -- start start 2024-03-29T04:08:54.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c5415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c5415bb10 0x7f7c5415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54158a80 0x7f7c54158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5406be40 con 0x7f7c40004800 2024-03-29T04:08:54.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c540b0f70 con 0x7f7c54158a80 2024-03-29T04:08:54.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c588a5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7c540ae160 con 0x7f7c5415bb10 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c5259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c5415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c5259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c5415b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45204/0 (socket says 172.21.15.156:45204) 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c5259c640 1 -- 172.21.15.156:0/1547985097 learned_addr learned my addr 172.21.15.156:0/1547985097 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c52d9d640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54158a80 0x7f7c54158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.251+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c5415bb10 0x7f7c5415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c5259c640 1 -- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c5415bb10 msgr2=0x7f7c5415bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c5259c640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c5415bb10 0x7f7c5415bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c5259c640 1 -- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54158a80 msgr2=0x7f7c54158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c5259c640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54158a80 0x7f7c54158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c5259c640 1 -- 172.21.15.156:0/1547985097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5415c680 con 0x7f7c40004800 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c5415bb10 0x7f7c5415bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c5259c640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c5415b0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400e460 tx=0x7f7c4400e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44019070 con 0x7f7c40004800 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c440052e0 con 0x7f7c40004800 2024-03-29T04:08:54.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44003830 con 0x7f7c40004800 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c540b3d00 con 0x7f7c40004800 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.252+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c540b42c0 con 0x7f7c40004800 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.253+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c54068d20 con 0x7f7c40004800 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.253+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7c4401c070 con 0x7f7c40004800 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.254+0000 7f7c4affd640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7c3406c530 0x7f7c3406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.254+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7c3406c530 0x7f7c3406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.254+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7c4408ab50 con 0x7f7c40004800 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.254+0000 7f7c4bfff640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7c3406c530 0x7f7c3406e9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c034000 tx=0x7f7c3c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:54.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.258+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7c44058bb0 con 0x7f7c40004800 2024-03-29T04:08:54.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.475+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c540b27d0 con 0x7f7c40004800 2024-03-29T04:08:54.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.476+0000 7f7c4affd640 1 -- 172.21.15.156:0/1547985097 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16642 (secure 0 0 0) 0x7f7c4405ceb0 con 0x7f7c40004800 2024-03-29T04:08:54.456 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:54.456 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":20,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:53.151435+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7c3406c530 msgr2=0x7f7c3406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7c3406c530 0x7f7c3406e9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c034000 tx=0x7f7c3c009030 comp rx=0 tx=0).stop 2024-03-29T04:08:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 msgr2=0x7f7c5415b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c5415b0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400e460 tx=0x7f7c4400e920 comp rx=0 tx=0).stop 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 shutdown_connections 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7c3406c530 0x7f7c3406e9f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54158a80 0x7f7c54158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c5415bb10 0x7f7c5415bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 --2- 172.21.15.156:0/1547985097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c40004800 0x7f7c5415b0c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.478+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 >> 172.21.15.156:0/1547985097 conn(0x7f7c5409aa40 msgr2=0x7f7c54141830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.479+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 shutdown_connections 2024-03-29T04:08:54.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.479+0000 7f7c588a5640 1 -- 172.21.15.156:0/1547985097 wait complete. 2024-03-29T04:08:54.469 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T04:08:54.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.594+0000 7f1f813e2640 1 Processor -- start 2024-03-29T04:08:54.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.594+0000 7f1f813e2640 1 -- start start 2024-03-29T04:08:54.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f813e2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 0x7f1f7c0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f813e2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1f7c0d2a00 0x7f1f7c0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f813e2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f7c0d0020 con 0x7f1f7c0cfb90 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f813e2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7c06c970 con 0x7f1f7c0d2a00 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f813e2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7c0d2fd0 con 0x7f1f7c15e540 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f7b7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1f7c0cfb90 0x7f1f7c0d0d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58944/0 (socket says 172.21.15.156:58944) 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f7b7fe640 1 -- 172.21.15.156:0/341563149 learned_addr learned my addr 172.21.15.156:0/341563149 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f7affd640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 0x7f1f7c0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f7a7fc640 1 --2- 172.21.15.156:0/341563149 >> v2:172.21.15.203:3300/0 conn(0x7f1f7c0d2a00 0x7f1f7c0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1718911655 0 0) 0x7f1f7c0d0020 con 0x7f1f7c0cfb90 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f68003520 con 0x7f1f7c0cfb90 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2613408502 0 0) 0x7f1f68003520 con 0x7f1f7c0cfb90 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 msgr2=0x7f1f7c0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 0x7f1f7c0cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 >> v2:172.21.15.203:3300/0 conn(0x7f1f7c0d2a00 msgr2=0x7f1f7c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 --2- 172.21.15.156:0/341563149 >> v2:172.21.15.203:3300/0 conn(0x7f1f7c0d2a00 0x7f1f7c0cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c0d32b0 con 0x7f1f7c0cfb90 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f1f6c001670 con 0x7f1f7c0cfb90 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 >> v1:172.21.15.156:6789/0 conn(0x7f1f7c0cfb90 legacy=0x7f1f7c0d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f680039b0 0x7f1f68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 0x7f1f68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f68004d40 0x7f1f680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c06c970 con 0x7f1f68004d40 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c0d2fd0 con 0x7f1f680039b0 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.595+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6800b9c0 con 0x7f1f7c15e540 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7b7fe640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f68004d40 0x7f1f680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7a7fc640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 0x7f1f68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7affd640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f680039b0 0x7f1f68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7b7fe640 1 -- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 msgr2=0x7f1f68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7b7fe640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c15e540 0x7f1f68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7b7fe640 1 -- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f680039b0 msgr2=0x7f1f68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7b7fe640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f680039b0 0x7f1f68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.596+0000 7f1f7b7fe640 1 -- 172.21.15.156:0/341563149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f68003520 con 0x7f1f68004d40 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f7b7fe640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f68004d40 0x7f1f680051c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00c240 tx=0x7f1f68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf07d4b19a5addd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c007d30 con 0x7f1f68004d40 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f6c030070 con 0x7f1f68004d40 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f79ffb640 1 -- 172.21.15.156:0/341563149 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c02cd60 con 0x7f1f68004d40 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f813e2640 1 -- 172.21.15.156:0/341563149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f68004d40 msgr2=0x7f1f680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f813e2640 1 --2- 172.21.15.156:0/341563149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f68004d40 0x7f1f680051c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00c240 tx=0x7f1f68005830 comp rx=0 tx=0).stop 2024-03-29T04:08:54.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f7b7fe640 1 -- 172.21.15.156:0/341563149 reap_dead start 2024-03-29T04:08:54.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f813e2640 1 -- 172.21.15.156:0/341563149 shutdown_connections 2024-03-29T04:08:54.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f813e2640 1 -- 172.21.15.156:0/341563149 >> 172.21.15.156:0/341563149 conn(0x7f1f7c0d4450 msgr2=0x7f1f7c0d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:54.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f813e2640 1 -- 172.21.15.156:0/341563149 shutdown_connections 2024-03-29T04:08:54.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.597+0000 7f1f813e2640 1 -- 172.21.15.156:0/341563149 wait complete. 2024-03-29T04:08:54.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 Processor -- start 2024-03-29T04:08:54.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 -- start start 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c153b00 0x7f1f7c0cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 0x7f1f7c1549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c154f20 0x7f1f7c0ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c15eaa0 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c15e920 con 0x7f1f7c153b00 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f813e2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7c06de10 con 0x7f1f7c154f20 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f7a7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 0x7f1f7c1549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f7a7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 0x7f1f7c1549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45228/0 (socket says 172.21.15.156:45228) 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.599+0000 7f1f7a7fc640 1 -- 172.21.15.156:0/3783359152 learned_addr learned my addr 172.21.15.156:0/3783359152 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7affd640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c153b00 0x7f1f7c0cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7b7fe640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c154f20 0x7f1f7c0ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7a7fc640 1 -- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c154f20 msgr2=0x7f1f7c0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7a7fc640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c154f20 0x7f1f7c0ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7a7fc640 1 -- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c153b00 msgr2=0x7f1f7c0cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7a7fc640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c153b00 0x7f1f7c0cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7a7fc640 1 -- 172.21.15.156:0/3783359152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c0c9930 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7affd640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c153b00 0x7f1f7c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7b7fe640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c154f20 0x7f1f7c0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f7a7fc640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 0x7f1f7c1549e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f64011a70 tx=0x7f1f64011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6400fb50 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f6400fcf0 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c0c9c10 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6400fea0 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.600+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c0c9600 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.601+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c0a23a0 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.602+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1f64012040 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.602+0000 7f1f737fe640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1f5c06c560 0x7f1f5c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:54.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.602+0000 7f1f7affd640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1f5c06c560 0x7f1f5c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:54.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.602+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1f6408be00 con 0x7f1f7c0cc9c0 2024-03-29T04:08:54.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.603+0000 7f1f7affd640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1f5c06c560 0x7f1f5c06ea20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f74007440 tx=0x7f1f7400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:54.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:54.606+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1f64019030 con 0x7f1f7c0cc9c0 2024-03-29T04:08:55.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.024+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1f7c0d1fd0 con 0x7f1f7c0cc9c0 2024-03-29T04:08:55.408 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:08:55.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:08:55.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.819+0000 7f1f737fe640 1 -- 172.21.15.156:0/3783359152 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 118+0+0 (secure 0 0 0) 0x7f1f6405a4b0 con 0x7f1f7c0cc9c0 2024-03-29T04:08:55.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1f5c06c560 msgr2=0x7f1f5c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:55.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1f5c06c560 0x7f1f5c06ea20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f74007440 tx=0x7f1f7400b040 comp rx=0 tx=0).stop 2024-03-29T04:08:55.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 msgr2=0x7f1f7c1549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:55.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 0x7f1f7c1549e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f64011a70 tx=0x7f1f64011f30 comp rx=0 tx=0).stop 2024-03-29T04:08:55.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 shutdown_connections 2024-03-29T04:08:55.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1f5c06c560 0x7f1f5c06ea20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f7c154f20 0x7f1f7c0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f7c0cc9c0 0x7f1f7c1549e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 --2- 172.21.15.156:0/3783359152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c153b00 0x7f1f7c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 >> 172.21.15.156:0/3783359152 conn(0x7f1f7c0d4450 msgr2=0x7f1f7c1616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:55.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 shutdown_connections 2024-03-29T04:08:55.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.821+0000 7f1f813e2640 1 -- 172.21.15.156:0/3783359152 wait complete. 2024-03-29T04:08:55.813 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T04:08:55.813 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T04:08:55.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.940+0000 7f77bd07e640 1 Processor -- start 2024-03-29T04:08:55.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.940+0000 7f77bd07e640 1 -- start start 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77bd07e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f77b815e960 0x7f77b806f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77bd07e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77bd07e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b80b5ed0 con 0x7f77b8070480 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77bd07e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f77b806bf60 con 0x7f77b815e960 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77bd07e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f77b809aa40 con 0x7f77b806b5b0 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b6d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f77b815e960 0x7f77b806f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:55.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77affff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f77b8070480 0x7f77b80b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58950/0 (socket says 172.21.15.156:58950) 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77affff640 1 -- 172.21.15.156:0/2064734418 learned_addr learned my addr 172.21.15.156:0/2064734418 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b759e640 1 --2- 172.21.15.156:0/2064734418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b6d9d640 1 -- 172.21.15.156:0/2064734418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 msgr2=0x7f77b806ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b6d9d640 1 --2- 172.21.15.156:0/2064734418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b806ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b6d9d640 1 -- 172.21.15.156:0/2064734418 >> v1:172.21.15.156:6789/0 conn(0x7f77b8070480 legacy=0x7f77b80b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b6d9d640 1 -- 172.21.15.156:0/2064734418 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b809ac90 con 0x7f77b815e960 2024-03-29T04:08:55.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.941+0000 7f77b659c640 1 -- 172.21.15.156:0/2064734418 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2649947649 0 0) 0x7f77b80b5ed0 con 0x7f77b8070480 2024-03-29T04:08:55.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.942+0000 7f77b6d9d640 1 --2- 172.21.15.156:0/2064734418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b806f900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f77a80097a0 tx=0x7f77a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedb9b8fec196ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.942+0000 7f77b659c640 1 -- 172.21.15.156:0/2064734418 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a8035020 con 0x7f77b815e960 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.942+0000 7f77b659c640 1 -- 172.21.15.156:0/2064734418 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a8031830 con 0x7f77b815e960 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.942+0000 7f77b659c640 1 -- 172.21.15.156:0/2064734418 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a8031c00 con 0x7f77b815e960 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 -- 172.21.15.156:0/2064734418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 msgr2=0x7f77b806f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2064734418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b806f900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f77a80097a0 tx=0x7f77a802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 -- 172.21.15.156:0/2064734418 shutdown_connections 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2064734418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b806ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2064734418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b806f900 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 -- 172.21.15.156:0/2064734418 >> 172.21.15.156:0/2064734418 conn(0x7f77b8072ce0 msgr2=0x7f77b80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 -- 172.21.15.156:0/2064734418 shutdown_connections 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.943+0000 7f77bd07e640 1 -- 172.21.15.156:0/2064734418 wait complete. 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.944+0000 7f77bd07e640 1 Processor -- start 2024-03-29T04:08:55.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.944+0000 7f77bd07e640 1 -- start start 2024-03-29T04:08:55.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77bd07e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b80b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:55.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77bd07e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b8070480 0x7f77b809c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:55.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77bd07e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b8152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77bd07e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77b80701f0 con 0x7f77b815e960 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77bd07e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77b80b6450 con 0x7f77b8070480 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77bd07e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f77b80b49e0 con 0x7f77b806b5b0 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b8152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b8152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45246/0 (socket says 172.21.15.156:45246) 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 -- 172.21.15.156:0/2040835459 learned_addr learned my addr 172.21.15.156:0/2040835459 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b6d9d640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b80b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77affff640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b8070480 0x7f77b809c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 -- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 msgr2=0x7f77b80b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b80b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 -- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b8070480 msgr2=0x7f77b809c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b8070480 0x7f77b809c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.945+0000 7f77b759e640 1 -- 172.21.15.156:0/2040835459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b809cc60 con 0x7f77b815e960 2024-03-29T04:08:55.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.946+0000 7f77b759e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b8152960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f77b0007e40 tx=0x7f77b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:55.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.946+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b0014620 con 0x7f77b815e960 2024-03-29T04:08:55.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.946+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b0019070 con 0x7f77b815e960 2024-03-29T04:08:55.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.946+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b809cf40 con 0x7f77b815e960 2024-03-29T04:08:55.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.946+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b0014a20 con 0x7f77b815e960 2024-03-29T04:08:55.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.946+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b81599f0 con 0x7f77b815e960 2024-03-29T04:08:55.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.947+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b8073370 con 0x7f77b815e960 2024-03-29T04:08:55.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.947+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f77b0014bc0 con 0x7f77b815e960 2024-03-29T04:08:55.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.948+0000 7f77af7fe640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f779806c530 0x7f779806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:55.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.948+0000 7f77b6d9d640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f779806c530 0x7f779806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:55.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.948+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f77b008bdb0 con 0x7f77b815e960 2024-03-29T04:08:55.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.948+0000 7f77b6d9d640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f779806c530 0x7f779806e9f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77a8036000 tx=0x7f77a803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:55.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:55.952+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f77b0059e10 con 0x7f77b815e960 2024-03-29T04:08:56.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.191+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f77b804f9d0 con 0x7f77b815e960 2024-03-29T04:08:56.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.819+0000 7f77af7fe640 1 -- 172.21.15.156:0/2040835459 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7f77b005e110 con 0x7f77b815e960 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f779806c530 msgr2=0x7f779806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f779806c530 0x7f779806e9f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77a8036000 tx=0x7f77a803c040 comp rx=0 tx=0).stop 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 msgr2=0x7f77b8152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b8152960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f77b0007e40 tx=0x7f77b000e4b0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 shutdown_connections 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f779806c530 0x7f779806e9f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77b815e960 0x7f77b8152960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b8070480 0x7f77b809c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 --2- 172.21.15.156:0/2040835459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77b806b5b0 0x7f77b80b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 >> 172.21.15.156:0/2040835459 conn(0x7f77b8072ce0 msgr2=0x7f77b815d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 shutdown_connections 2024-03-29T04:08:56.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.822+0000 7f77bd07e640 1 -- 172.21.15.156:0/2040835459 wait complete. 2024-03-29T04:08:56.812 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T04:08:56.812 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:08:56.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.934+0000 7f3bdc372640 1 Processor -- start 2024-03-29T04:08:56.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.934+0000 7f3bdc372640 1 -- start start 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bdc372640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 0x7f3bd415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bdc372640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 0x7f3bd406f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bdc372640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bd415c7e0 con 0x7f3bd406f5d0 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bdc372640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3bd4072760 con 0x7f3bd40af6b0 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bdc372640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd40afa90 con 0x7f3bd40b3050 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 0x7f3bd406f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:56.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bda10e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 0x7f3bd415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 0x7f3bd406f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42404/0 (socket says 172.21.15.156:42404) 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 -- 172.21.15.156:0/3924810376 learned_addr learned my addr 172.21.15.156:0/3924810376 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bda10e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 0x7f3bd415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35054/0 (socket says 172.21.15.156:35054) 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd910c640 1 -- 172.21.15.156:0/3924810376 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667888869 0 0) 0x7f3bcc001490 con 0x7f3bd406f5d0 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd910c640 1 -- 172.21.15.156:0/3924810376 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bc0003520 con 0x7f3bd406f5d0 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 -- 172.21.15.156:0/3924810376 >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 msgr2=0x7f3bd415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 --2- 172.21.15.156:0/3924810376 >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 0x7f3bd415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 -- 172.21.15.156:0/3924810376 >> v1:172.21.15.203:6789/0 conn(0x7f3bd406f5d0 legacy=0x7f3bd415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bd990d640 1 -- 172.21.15.156:0/3924810376 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd40b3510 con 0x7f3bd40b3050 2024-03-29T04:08:56.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.935+0000 7f3bda10e640 1 --2- 172.21.15.156:0/3924810376 >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 0x7f3bd415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.936+0000 7f3bd990d640 1 --2- 172.21.15.156:0/3924810376 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 0x7f3bd406f090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400bb10 tx=0x7f3bc402e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a78596c3cb5c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.936+0000 7f3bd910c640 1 -- 172.21.15.156:0/3924810376 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc403b070 con 0x7f3bd40b3050 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.936+0000 7f3bd910c640 1 -- 172.21.15.156:0/3924810376 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc4036440 con 0x7f3bd40b3050 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.936+0000 7f3bd910c640 1 -- 172.21.15.156:0/3924810376 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc40367f0 con 0x7f3bd40b3050 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 -- 172.21.15.156:0/3924810376 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 msgr2=0x7f3bd406f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 --2- 172.21.15.156:0/3924810376 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 0x7f3bd406f090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400bb10 tx=0x7f3bc402e0c0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 -- 172.21.15.156:0/3924810376 shutdown_connections 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 --2- 172.21.15.156:0/3924810376 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40b3050 0x7f3bd406f090 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 --2- 172.21.15.156:0/3924810376 >> v2:172.21.15.203:3300/0 conn(0x7f3bd40af6b0 0x7f3bd415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 -- 172.21.15.156:0/3924810376 >> 172.21.15.156:0/3924810376 conn(0x7f3bd40b1d50 msgr2=0x7f3bd40b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:56.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 -- 172.21.15.156:0/3924810376 shutdown_connections 2024-03-29T04:08:56.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.937+0000 7f3bdc372640 1 -- 172.21.15.156:0/3924810376 wait complete. 2024-03-29T04:08:56.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.938+0000 7f3bdc372640 1 Processor -- start 2024-03-29T04:08:56.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.938+0000 7f3bdc372640 1 -- start start 2024-03-29T04:08:56.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bdc372640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bd406f5d0 0x7f3bd41e5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:56.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bdc372640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40af6b0 0x7f3bd4141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bdc372640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 0x7f3bd4141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bdc372640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd40b3db0 con 0x7f3bd40b3050 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bdc372640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd40b3ad0 con 0x7f3bd406f5d0 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bdc372640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd40b4090 con 0x7f3bd40af6b0 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 0x7f3bd4141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 0x7f3bd4141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45262/0 (socket says 172.21.15.156:45262) 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 -- 172.21.15.156:0/907638802 learned_addr learned my addr 172.21.15.156:0/907638802 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda10e640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bd406f5d0 0x7f3bd41e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:56.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bd990d640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40af6b0 0x7f3bd4141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:56.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 -- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40af6b0 msgr2=0x7f3bd4141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:56.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40af6b0 0x7f3bd4141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 -- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bd406f5d0 msgr2=0x7f3bd41e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:56.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bd406f5d0 0x7f3bd41e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:56.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.939+0000 7f3bda90f640 1 -- 172.21.15.156:0/907638802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd41e3b50 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.940+0000 7f3bda90f640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 0x7f3bd4141690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc002230 tx=0x7f3bcc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.940+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc003b70 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.940+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bcc003d10 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.940+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd41e3e30 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.940+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc005310 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.940+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd41e43f0 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.941+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd4068610 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.941+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3bcc0056c0 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.942+0000 7f3bcaffd640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3bb806c530 0x7f3bb806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.942+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3bcc08b7e0 con 0x7f3bd40b3050 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.942+0000 7f3bda10e640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3bb806c530 0x7f3bb806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:56.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.942+0000 7f3bda10e640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3bb806c530 0x7f3bb806e9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009560 tx=0x7f3bd0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:56.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:56.946+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3bd415c7e0 con 0x7f3bd40b3050 2024-03-29T04:08:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.165+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bd406da70 con 0x7f3bd40b3050 2024-03-29T04:08:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.165+0000 7f3bcaffd640 1 -- 172.21.15.156:0/907638802 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f3bcc0597b0 con 0x7f3bd40b3050 2024-03-29T04:08:57.145 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:57.145 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:08:57.146 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:08:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3bb806c530 msgr2=0x7f3bb806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3bb806c530 0x7f3bb806e9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009560 tx=0x7f3bd0008040 comp rx=0 tx=0).stop 2024-03-29T04:08:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 msgr2=0x7f3bd4141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 0x7f3bd4141690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc002230 tx=0x7f3bcc00cb90 comp rx=0 tx=0).stop 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 shutdown_connections 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3bb806c530 0x7f3bb806e9f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bd40b3050 0x7f3bd4141690 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3bd40af6b0 0x7f3bd4141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.167+0000 7f3bdc372640 1 --2- 172.21.15.156:0/907638802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bd406f5d0 0x7f3bd41e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.168+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 >> 172.21.15.156:0/907638802 conn(0x7f3bd40b1d50 msgr2=0x7f3bd40cfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.168+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 shutdown_connections 2024-03-29T04:08:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.168+0000 7f3bdc372640 1 -- 172.21.15.156:0/907638802 wait complete. 2024-03-29T04:08:57.158 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:57.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.280+0000 7faca0560640 1 Processor -- start 2024-03-29T04:08:57.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.280+0000 7faca0560640 1 -- start start 2024-03-29T04:08:57.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7faca0560640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98070ad0 0x7fac98160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7faca0560640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fac98162230 0x7fac9809b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7faca0560640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac9806fc80 con 0x7fac9815c4f0 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7faca0560640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fac981627a0 con 0x7fac98162230 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7faca0560640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fac98162a80 con 0x7fac98070ad0 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7fac9eafd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fac98162230 0x7fac9809b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7fac9e2fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fac9815c4f0 0x7fac980703c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58956/0 (socket says 172.21.15.156:58956) 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.281+0000 7fac9e2fc640 1 -- 172.21.15.156:0/2609351204 learned_addr learned my addr 172.21.15.156:0/2609351204 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.282+0000 7fac9dafb640 1 --2- 172.21.15.156:0/2609351204 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98070ad0 0x7fac98160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.282+0000 7fac9eafd640 1 -- 172.21.15.156:0/2609351204 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98070ad0 msgr2=0x7fac98160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.282+0000 7fac9eafd640 1 --2- 172.21.15.156:0/2609351204 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98070ad0 0x7fac98160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.282+0000 7fac9eafd640 1 -- 172.21.15.156:0/2609351204 >> v1:172.21.15.156:6789/0 conn(0x7fac9815c4f0 legacy=0x7fac980703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.282+0000 7fac9d2fa640 1 -- 172.21.15.156:0/2609351204 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090527184 0 0) 0x7fac9806fc80 con 0x7fac9815c4f0 2024-03-29T04:08:57.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.282+0000 7fac9eafd640 1 -- 172.21.15.156:0/2609351204 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac98162d60 con 0x7fac98162230 2024-03-29T04:08:57.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7fac9eafd640 1 --2- 172.21.15.156:0/2609351204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac98162230 0x7fac9809b420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac90009d90 tx=0x7fac9002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2964f4a6a0609963 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:57.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7fac9d2fa640 1 -- 172.21.15.156:0/2609351204 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac90039070 con 0x7fac98162230 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7fac9d2fa640 1 -- 172.21.15.156:0/2609351204 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9002cc50 con 0x7fac98162230 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7fac9d2fa640 1 -- 172.21.15.156:0/2609351204 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac90034440 con 0x7fac98162230 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 -- 172.21.15.156:0/2609351204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac98162230 msgr2=0x7fac9809b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 --2- 172.21.15.156:0/2609351204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac98162230 0x7fac9809b420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac90009d90 tx=0x7fac9002bea0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 -- 172.21.15.156:0/2609351204 shutdown_connections 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 --2- 172.21.15.156:0/2609351204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac98162230 0x7fac9809b420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 --2- 172.21.15.156:0/2609351204 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98070ad0 0x7fac98160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 -- 172.21.15.156:0/2609351204 >> 172.21.15.156:0/2609351204 conn(0x7fac980d2250 msgr2=0x7fac980d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 -- 172.21.15.156:0/2609351204 shutdown_connections 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.283+0000 7faca0560640 1 -- 172.21.15.156:0/2609351204 wait complete. 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.285+0000 7faca0560640 1 Processor -- start 2024-03-29T04:08:57.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.285+0000 7faca0560640 1 -- start start 2024-03-29T04:08:57.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7faca0560640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac98070ad0 0x7fac9814d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7faca0560640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 0x7fac980b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7faca0560640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98162230 0x7fac9815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7faca0560640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac980d05a0 con 0x7fac9815c4f0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7faca0560640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac980d0420 con 0x7fac98070ad0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7faca0560640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fac980b1160 con 0x7fac98162230 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7fac9dafb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 0x7fac980b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7fac9dafb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 0x7fac980b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45294/0 (socket says 172.21.15.156:45294) 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.286+0000 7fac9dafb640 1 -- 172.21.15.156:0/2472084409 learned_addr learned my addr 172.21.15.156:0/2472084409 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9e2fc640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac98070ad0 0x7fac9814d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9eafd640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98162230 0x7fac9815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9dafb640 1 -- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98162230 msgr2=0x7fac9815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9dafb640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98162230 0x7fac9815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9dafb640 1 -- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac98070ad0 msgr2=0x7fac9814d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9dafb640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac98070ad0 0x7fac9814d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9dafb640 1 -- 172.21.15.156:0/2472084409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9814f120 con 0x7fac9815c4f0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac9dafb640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 0x7fac980b7790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fac8800e9e0 tx=0x7fac8800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88017070 con 0x7fac9815c4f0 2024-03-29T04:08:57.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8800c970 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9814f400 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8800cd60 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.287+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac980b7cd0 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.288+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac98068610 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.288+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fac8801a070 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.289+0000 7fac8effd640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fac7c06c530 0x7fac7c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.289+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac8808b690 con 0x7fac9815c4f0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.289+0000 7fac9e2fc640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fac7c06c530 0x7fac7c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.289+0000 7fac9e2fc640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fac7c06c530 0x7fac7c06e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fac94007fd0 tx=0x7fac94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:57.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.293+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fac88059cb0 con 0x7fac9815c4f0 2024-03-29T04:08:57.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.578+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac980d3160 con 0x7fac9815c4f0 2024-03-29T04:08:57.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.579+0000 7fac8effd640 1 -- 172.21.15.156:0/2472084409 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fac880595b0 con 0x7fac9815c4f0 2024-03-29T04:08:57.559 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:57.559 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fac7c06c530 msgr2=0x7fac7c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fac7c06c530 0x7fac7c06e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fac94007fd0 tx=0x7fac94009040 comp rx=0 tx=0).stop 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 msgr2=0x7fac980b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 0x7fac980b7790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fac8800e9e0 tx=0x7fac8800eea0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 shutdown_connections 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fac7c06c530 0x7fac7c06e9f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac98162230 0x7fac9815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9815c4f0 0x7fac980b7790 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 --2- 172.21.15.156:0/2472084409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac98070ad0 0x7fac9814d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 >> 172.21.15.156:0/2472084409 conn(0x7fac980d2250 msgr2=0x7fac980d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:57.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 shutdown_connections 2024-03-29T04:08:57.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.581+0000 7faca0560640 1 -- 172.21.15.156:0/2472084409 wait complete. 2024-03-29T04:08:57.572 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T04:08:57.572 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:08:57.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.698+0000 7f3d37c14640 1 Processor -- start 2024-03-29T04:08:57.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d37c14640 1 -- start start 2024-03-29T04:08:57.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d37c14640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3d3015b240 0x7f3d300b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d37c14640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d37c14640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d300afa60 con 0x7f3d300b03d0 2024-03-29T04:08:57.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d37c14640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3d300b0810 con 0x7f3d3015b240 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d37c14640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3d300af4a0 con 0x7f3d3015b620 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d359b0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3d3015b240 0x7f3d300b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d361b1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f3d300b03d0 0x7f3d300ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58958/0 (socket says 172.21.15.156:58958) 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.699+0000 7f3d361b1640 1 -- 172.21.15.156:0/3335523098 learned_addr learned my addr 172.21.15.156:0/3335523098 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.700+0000 7f3d359b0640 1 -- 172.21.15.156:0/3335523098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 msgr2=0x7f3d300afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.700+0000 7f3d351af640 1 --2- 172.21.15.156:0/3335523098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.700+0000 7f3d359b0640 1 --2- 172.21.15.156:0/3335523098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.700+0000 7f3d359b0640 1 -- 172.21.15.156:0/3335523098 >> v1:172.21.15.156:6789/0 conn(0x7f3d300b03d0 legacy=0x7f3d300ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.700+0000 7f3d349ae640 1 -- 172.21.15.156:0/3335523098 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552294117 0 0) 0x7f3d300afa60 con 0x7f3d300b03d0 2024-03-29T04:08:57.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.700+0000 7f3d359b0640 1 -- 172.21.15.156:0/3335523098 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d300af780 con 0x7f3d3015b240 2024-03-29T04:08:57.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.701+0000 7f3d359b0640 1 --2- 172.21.15.156:0/3335523098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d300b0a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d200097a0 tx=0x7f3d2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa25a90c946815d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:57.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.701+0000 7f3d349ae640 1 -- 172.21.15.156:0/3335523098 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20035020 con 0x7f3d3015b240 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.701+0000 7f3d349ae640 1 -- 172.21.15.156:0/3335523098 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d20031830 con 0x7f3d3015b240 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.701+0000 7f3d349ae640 1 -- 172.21.15.156:0/3335523098 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20031c00 con 0x7f3d3015b240 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 -- 172.21.15.156:0/3335523098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 msgr2=0x7f3d300b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 --2- 172.21.15.156:0/3335523098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d300b0a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d200097a0 tx=0x7f3d2002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 -- 172.21.15.156:0/3335523098 shutdown_connections 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 --2- 172.21.15.156:0/3335523098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 --2- 172.21.15.156:0/3335523098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d300b0a10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 -- 172.21.15.156:0/3335523098 >> 172.21.15.156:0/3335523098 conn(0x7f3d3009aa40 msgr2=0x7f3d300b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 -- 172.21.15.156:0/3335523098 shutdown_connections 2024-03-29T04:08:57.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.702+0000 7f3d37c14640 1 -- 172.21.15.156:0/3335523098 wait complete. 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.703+0000 7f3d37c14640 1 Processor -- start 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.703+0000 7f3d37c14640 1 -- start start 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d300b03d0 0x7f3d30153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d3009b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300aedc0 con 0x7f3d3015b240 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300b43e0 con 0x7f3d300b03d0 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3d300aef40 con 0x7f3d3015b620 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d3009b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d3009b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45306/0 (socket says 172.21.15.156:45306) 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 -- 172.21.15.156:0/1378802072 learned_addr learned my addr 172.21.15.156:0/1378802072 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d361b1640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d359b0640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d300b03d0 0x7f3d30153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 -- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 msgr2=0x7f3d300b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 -- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d300b03d0 msgr2=0x7f3d30153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d300b03d0 0x7f3d30153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 -- 172.21.15.156:0/1378802072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3009e3b0 con 0x7f3d3015b240 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d361b1640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d359b0640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d300b03d0 0x7f3d30153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d351af640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d3009b1c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00c7c0 tx=0x7f3d2c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c015070 con 0x7f3d3015b240 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c007820 con 0x7f3d3015b240 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d30153b70 con 0x7f3d3015b240 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c007bd0 con 0x7f3d3015b240 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.704+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3009e910 con 0x7f3d3015b240 2024-03-29T04:08:57.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.705+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d30068d20 con 0x7f3d3015b240 2024-03-29T04:08:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.706+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3d2c018070 con 0x7f3d3015b240 2024-03-29T04:08:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.706+0000 7f3d267fc640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3d1406c530 0x7f3d1406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.706+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3d2c08b1c0 con 0x7f3d3015b240 2024-03-29T04:08:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.706+0000 7f3d359b0640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3d1406c530 0x7f3d1406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.707+0000 7f3d359b0640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3d1406c530 0x7f3d1406e9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3d20036000 tx=0x7f3d20008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:57.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.715+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3d3009e3b0 con 0x7f3d3015b240 2024-03-29T04:08:57.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.895+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d3009b700 con 0x7f3d1406c530 2024-03-29T04:08:57.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.901+0000 7f3d267fc640 1 -- 172.21.15.156:0/1378802072 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251933 (secure 0 0 0) 0x7f3d3009b700 con 0x7f3d1406c530 2024-03-29T04:08:57.883 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:57.885 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:08:57.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3d1406c530 msgr2=0x7f3d1406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3d1406c530 0x7f3d1406e9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3d20036000 tx=0x7f3d20008030 comp rx=0 tx=0).stop 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 msgr2=0x7f3d3009b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d3009b1c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00c7c0 tx=0x7f3d2c00cc80 comp rx=0 tx=0).stop 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 shutdown_connections 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3d1406c530 0x7f3d1406e9f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3d3015b620 0x7f3d300b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3015b240 0x7f3d3009b1c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 --2- 172.21.15.156:0/1378802072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d300b03d0 0x7f3d30153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 >> 172.21.15.156:0/1378802072 conn(0x7f3d3009aa40 msgr2=0x7f3d300d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 shutdown_connections 2024-03-29T04:08:57.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:57.904+0000 7f3d37c14640 1 -- 172.21.15.156:0/1378802072 wait complete. 2024-03-29T04:08:57.894 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-29T04:08:57.637924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235664,"kb_used_data":2840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739056,"statfs":{"total":773094113280,"available":772852793344,"internally_reserved":0,"allocated":2908160,"data_stored":1756236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.848325"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546238+0000","last_change":"2024-03-29T04:08:53.827540+0000","last_active":"2024-03-29T04:08:55.546238+0000","last_peered":"2024-03-29T04:08:55.546238+0000","last_clean":"2024-03-29T04:08:55.546238+0000","last_became_active":"2024-03-29T04:08:53.827301+0000","last_became_peered":"2024-03-29T04:08:53.827301+0000","last_unstale":"2024-03-29T04:08:55.546238+0000","last_undegraded":"2024-03-29T04:08:55.546238+0000","last_fullsized":"2024-03-29T04:08:55.546238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812222+0000","last_change":"2024-03-29T04:08:53.809513+0000","last_active":"2024-03-29T04:08:54.812222+0000","last_peered":"2024-03-29T04:08:54.812222+0000","last_clean":"2024-03-29T04:08:54.812222+0000","last_became_active":"2024-03-29T04:08:53.809236+0000","last_became_peered":"2024-03-29T04:08:53.809236+0000","last_unstale":"2024-03-29T04:08:54.812222+0000","last_undegraded":"2024-03-29T04:08:54.812222+0000","last_fullsized":"2024-03-29T04:08:54.812222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823581+0000","last_change":"2024-03-29T04:08:53.821233+0000","last_active":"2024-03-29T04:08:53.823581+0000","last_peered":"2024-03-29T04:08:53.823581+0000","last_clean":"2024-03-29T04:08:53.823581+0000","last_became_active":"2024-03-29T04:08:53.820557+0000","last_became_peered":"2024-03-29T04:08:53.820557+0000","last_unstale":"2024-03-29T04:08:53.823581+0000","last_undegraded":"2024-03-29T04:08:53.823581+0000","last_fullsized":"2024-03-29T04:08:53.823581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814954+0000","last_change":"2024-03-29T04:08:53.824851+0000","last_active":"2024-03-29T04:08:54.814954+0000","last_peered":"2024-03-29T04:08:54.814954+0000","last_clean":"2024-03-29T04:08:54.814954+0000","last_became_active":"2024-03-29T04:08:53.824563+0000","last_became_peered":"2024-03-29T04:08:53.824563+0000","last_unstale":"2024-03-29T04:08:54.814954+0000","last_undegraded":"2024-03-29T04:08:54.814954+0000","last_fullsized":"2024-03-29T04:08:54.814954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546094+0000","last_change":"2024-03-29T04:08:53.820567+0000","last_active":"2024-03-29T04:08:55.546094+0000","last_peered":"2024-03-29T04:08:55.546094+0000","last_clean":"2024-03-29T04:08:55.546094+0000","last_became_active":"2024-03-29T04:08:53.820290+0000","last_became_peered":"2024-03-29T04:08:53.820290+0000","last_unstale":"2024-03-29T04:08:55.546094+0000","last_undegraded":"2024-03-29T04:08:55.546094+0000","last_fullsized":"2024-03-29T04:08:55.546094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814434+0000","last_change":"2024-03-29T04:08:53.818966+0000","last_active":"2024-03-29T04:08:54.814434+0000","last_peered":"2024-03-29T04:08:54.814434+0000","last_clean":"2024-03-29T04:08:54.814434+0000","last_became_active":"2024-03-29T04:08:53.818396+0000","last_became_peered":"2024-03-29T04:08:53.818396+0000","last_unstale":"2024-03-29T04:08:54.814434+0000","last_undegraded":"2024-03-29T04:08:54.814434+0000","last_fullsized":"2024-03-29T04:08:54.814434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545835+0000","last_change":"2024-03-29T04:08:53.817600+0000","last_active":"2024-03-29T04:08:55.545835+0000","last_peered":"2024-03-29T04:08:55.545835+0000","last_clean":"2024-03-29T04:08:55.545835+0000","last_became_active":"2024-03-29T04:08:53.817308+0000","last_became_peered":"2024-03-29T04:08:53.817308+0000","last_unstale":"2024-03-29T04:08:55.545835+0000","last_undegraded":"2024-03-29T04:08:55.545835+0000","last_fullsized":"2024-03-29T04:08:55.545835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.827597+0000","last_change":"2024-03-29T04:08:53.826247+0000","last_active":"2024-03-29T04:08:53.827597+0000","last_peered":"2024-03-29T04:08:53.827597+0000","last_clean":"2024-03-29T04:08:53.827597+0000","last_became_active":"2024-03-29T04:08:53.825974+0000","last_became_peered":"2024-03-29T04:08:53.825974+0000","last_unstale":"2024-03-29T04:08:53.827597+0000","last_undegraded":"2024-03-29T04:08:53.827597+0000","last_fullsized":"2024-03-29T04:08:53.827597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546330+0000","last_change":"2024-03-29T04:08:53.824041+0000","last_active":"2024-03-29T04:08:55.546330+0000","last_peered":"2024-03-29T04:08:55.546330+0000","last_clean":"2024-03-29T04:08:55.546330+0000","last_became_active":"2024-03-29T04:08:53.823788+0000","last_became_peered":"2024-03-29T04:08:53.823788+0000","last_unstale":"2024-03-29T04:08:55.546330+0000","last_undegraded":"2024-03-29T04:08:55.546330+0000","last_fullsized":"2024-03-29T04:08:55.546330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816424+0000","last_change":"2024-03-29T04:08:53.830205+0000","last_active":"2024-03-29T04:08:54.816424+0000","last_peered":"2024-03-29T04:08:54.816424+0000","last_clean":"2024-03-29T04:08:54.816424+0000","last_became_active":"2024-03-29T04:08:53.829931+0000","last_became_peered":"2024-03-29T04:08:53.829931+0000","last_unstale":"2024-03-29T04:08:54.816424+0000","last_undegraded":"2024-03-29T04:08:54.816424+0000","last_fullsized":"2024-03-29T04:08:54.816424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816205+0000","last_change":"2024-03-29T04:08:53.821458+0000","last_active":"2024-03-29T04:08:54.816205+0000","last_peered":"2024-03-29T04:08:54.816205+0000","last_clean":"2024-03-29T04:08:54.816205+0000","last_became_active":"2024-03-29T04:08:53.821168+0000","last_became_peered":"2024-03-29T04:08:53.821168+0000","last_unstale":"2024-03-29T04:08:54.816205+0000","last_undegraded":"2024-03-29T04:08:54.816205+0000","last_fullsized":"2024-03-29T04:08:54.816205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814279+0000","last_change":"2024-03-29T04:08:53.827058+0000","last_active":"2024-03-29T04:08:54.814279+0000","last_peered":"2024-03-29T04:08:54.814279+0000","last_clean":"2024-03-29T04:08:54.814279+0000","last_became_active":"2024-03-29T04:08:53.826167+0000","last_became_peered":"2024-03-29T04:08:53.826167+0000","last_unstale":"2024-03-29T04:08:54.814279+0000","last_undegraded":"2024-03-29T04:08:54.814279+0000","last_fullsized":"2024-03-29T04:08:54.814279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831543+0000","last_change":"2024-03-29T04:08:53.830239+0000","last_active":"2024-03-29T04:08:53.831543+0000","last_peered":"2024-03-29T04:08:53.831543+0000","last_clean":"2024-03-29T04:08:53.831543+0000","last_became_active":"2024-03-29T04:08:53.829295+0000","last_became_peered":"2024-03-29T04:08:53.829295+0000","last_unstale":"2024-03-29T04:08:53.831543+0000","last_undegraded":"2024-03-29T04:08:53.831543+0000","last_fullsized":"2024-03-29T04:08:53.831543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:39.590764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823281+0000","last_change":"2024-03-29T04:08:53.819700+0000","last_active":"2024-03-29T04:08:53.823281+0000","last_peered":"2024-03-29T04:08:53.823281+0000","last_clean":"2024-03-29T04:08:53.823281+0000","last_became_active":"2024-03-29T04:08:53.819174+0000","last_became_peered":"2024-03-29T04:08:53.819174+0000","last_unstale":"2024-03-29T04:08:53.823281+0000","last_undegraded":"2024-03-29T04:08:53.823281+0000","last_fullsized":"2024-03-29T04:08:53.823281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811825+0000","last_change":"2024-03-29T04:08:53.812928+0000","last_active":"2024-03-29T04:08:54.811825+0000","last_peered":"2024-03-29T04:08:54.811825+0000","last_clean":"2024-03-29T04:08:54.811825+0000","last_became_active":"2024-03-29T04:08:53.812536+0000","last_became_peered":"2024-03-29T04:08:53.812536+0000","last_unstale":"2024-03-29T04:08:54.811825+0000","last_undegraded":"2024-03-29T04:08:54.811825+0000","last_fullsized":"2024-03-29T04:08:54.811825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814066+0000","last_change":"2024-03-29T04:08:53.823468+0000","last_active":"2024-03-29T04:08:54.814066+0000","last_peered":"2024-03-29T04:08:54.814066+0000","last_clean":"2024-03-29T04:08:54.814066+0000","last_became_active":"2024-03-29T04:08:53.822972+0000","last_became_peered":"2024-03-29T04:08:53.822972+0000","last_unstale":"2024-03-29T04:08:54.814066+0000","last_undegraded":"2024-03-29T04:08:54.814066+0000","last_fullsized":"2024-03-29T04:08:54.814066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815690+0000","last_change":"2024-03-29T04:08:53.816411+0000","last_active":"2024-03-29T04:08:54.815690+0000","last_peered":"2024-03-29T04:08:54.815690+0000","last_clean":"2024-03-29T04:08:54.815690+0000","last_became_active":"2024-03-29T04:08:53.816126+0000","last_became_peered":"2024-03-29T04:08:53.816126+0000","last_unstale":"2024-03-29T04:08:54.815690+0000","last_undegraded":"2024-03-29T04:08:54.815690+0000","last_fullsized":"2024-03-29T04:08:54.815690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.825587+0000","last_change":"2024-03-29T04:08:53.824360+0000","last_active":"2024-03-29T04:08:53.825587+0000","last_peered":"2024-03-29T04:08:53.825587+0000","last_clean":"2024-03-29T04:08:53.825587+0000","last_became_active":"2024-03-29T04:08:53.823207+0000","last_became_peered":"2024-03-29T04:08:53.823207+0000","last_unstale":"2024-03-29T04:08:53.825587+0000","last_undegraded":"2024-03-29T04:08:53.825587+0000","last_fullsized":"2024-03-29T04:08:53.825587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546389+0000","last_change":"2024-03-29T04:08:53.822584+0000","last_active":"2024-03-29T04:08:55.546389+0000","last_peered":"2024-03-29T04:08:55.546389+0000","last_clean":"2024-03-29T04:08:55.546389+0000","last_became_active":"2024-03-29T04:08:53.821230+0000","last_became_peered":"2024-03-29T04:08:53.821230+0000","last_unstale":"2024-03-29T04:08:55.546389+0000","last_undegraded":"2024-03-29T04:08:55.546389+0000","last_fullsized":"2024-03-29T04:08:55.546389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816186+0000","last_change":"2024-03-29T04:08:53.828332+0000","last_active":"2024-03-29T04:08:54.816186+0000","last_peered":"2024-03-29T04:08:54.816186+0000","last_clean":"2024-03-29T04:08:54.816186+0000","last_became_active":"2024-03-29T04:08:53.827954+0000","last_became_peered":"2024-03-29T04:08:53.827954+0000","last_unstale":"2024-03-29T04:08:54.816186+0000","last_undegraded":"2024-03-29T04:08:54.816186+0000","last_fullsized":"2024-03-29T04:08:54.816186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823715+0000","last_change":"2024-03-29T04:08:53.821853+0000","last_active":"2024-03-29T04:08:53.823715+0000","last_peered":"2024-03-29T04:08:53.823715+0000","last_clean":"2024-03-29T04:08:53.823715+0000","last_became_active":"2024-03-29T04:08:53.820469+0000","last_became_peered":"2024-03-29T04:08:53.820469+0000","last_unstale":"2024-03-29T04:08:53.823715+0000","last_undegraded":"2024-03-29T04:08:53.823715+0000","last_fullsized":"2024-03-29T04:08:53.823715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814489+0000","last_change":"2024-03-29T04:08:53.818641+0000","last_active":"2024-03-29T04:08:54.814489+0000","last_peered":"2024-03-29T04:08:54.814489+0000","last_clean":"2024-03-29T04:08:54.814489+0000","last_became_active":"2024-03-29T04:08:53.818082+0000","last_became_peered":"2024-03-29T04:08:53.818082+0000","last_unstale":"2024-03-29T04:08:54.814489+0000","last_undegraded":"2024-03-29T04:08:54.814489+0000","last_fullsized":"2024-03-29T04:08:54.814489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823823+0000","last_change":"2024-03-29T04:08:53.816408+0000","last_active":"2024-03-29T04:08:53.823823+0000","last_peered":"2024-03-29T04:08:53.823823+0000","last_clean":"2024-03-29T04:08:53.823823+0000","last_became_active":"2024-03-29T04:08:53.816127+0000","last_became_peered":"2024-03-29T04:08:53.816127+0000","last_unstale":"2024-03-29T04:08:53.823823+0000","last_undegraded":"2024-03-29T04:08:53.823823+0000","last_fullsized":"2024-03-29T04:08:53.823823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:18.995986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814214+0000","last_change":"2024-03-29T04:08:53.820680+0000","last_active":"2024-03-29T04:08:54.814214+0000","last_peered":"2024-03-29T04:08:54.814214+0000","last_clean":"2024-03-29T04:08:54.814214+0000","last_became_active":"2024-03-29T04:08:53.819878+0000","last_became_peered":"2024-03-29T04:08:53.819878+0000","last_unstale":"2024-03-29T04:08:54.814214+0000","last_undegraded":"2024-03-29T04:08:54.814214+0000","last_fullsized":"2024-03-29T04:08:54.814214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.819987+0000","last_change":"2024-03-29T04:08:53.816044+0000","last_active":"2024-03-29T04:08:53.819987+0000","last_peered":"2024-03-29T04:08:53.819987+0000","last_clean":"2024-03-29T04:08:53.819987+0000","last_became_active":"2024-03-29T04:08:53.815744+0000","last_became_peered":"2024-03-29T04:08:53.815744+0000","last_unstale":"2024-03-29T04:08:53.819987+0000","last_undegraded":"2024-03-29T04:08:53.819987+0000","last_fullsized":"2024-03-29T04:08:53.819987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:16.901767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816521+0000","last_change":"2024-03-29T04:08:53.830594+0000","last_active":"2024-03-29T04:08:54.816521+0000","last_peered":"2024-03-29T04:08:54.816521+0000","last_clean":"2024-03-29T04:08:54.816521+0000","last_became_active":"2024-03-29T04:08:53.830266+0000","last_became_peered":"2024-03-29T04:08:53.830266+0000","last_unstale":"2024-03-29T04:08:54.816521+0000","last_undegraded":"2024-03-29T04:08:54.816521+0000","last_fullsized":"2024-03-29T04:08:54.816521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812076+0000","last_change":"2024-03-29T04:08:53.817458+0000","last_active":"2024-03-29T04:08:54.812076+0000","last_peered":"2024-03-29T04:08:54.812076+0000","last_clean":"2024-03-29T04:08:54.812076+0000","last_became_active":"2024-03-29T04:08:53.817179+0000","last_became_peered":"2024-03-29T04:08:53.817179+0000","last_unstale":"2024-03-29T04:08:54.812076+0000","last_undegraded":"2024-03-29T04:08:54.812076+0000","last_fullsized":"2024-03-29T04:08:54.812076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823651+0000","last_change":"2024-03-29T04:08:53.819794+0000","last_active":"2024-03-29T04:08:53.823651+0000","last_peered":"2024-03-29T04:08:53.823651+0000","last_clean":"2024-03-29T04:08:53.823651+0000","last_became_active":"2024-03-29T04:08:53.819541+0000","last_became_peered":"2024-03-29T04:08:53.819541+0000","last_unstale":"2024-03-29T04:08:53.823651+0000","last_undegraded":"2024-03-29T04:08:53.823651+0000","last_fullsized":"2024-03-29T04:08:53.823651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823889+0000","last_change":"2024-03-29T04:08:53.823205+0000","last_active":"2024-03-29T04:08:53.823889+0000","last_peered":"2024-03-29T04:08:53.823889+0000","last_clean":"2024-03-29T04:08:53.823889+0000","last_became_active":"2024-03-29T04:08:53.822900+0000","last_became_peered":"2024-03-29T04:08:53.822900+0000","last_unstale":"2024-03-29T04:08:53.823889+0000","last_undegraded":"2024-03-29T04:08:53.823889+0000","last_fullsized":"2024-03-29T04:08:53.823889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:39.294386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814624+0000","last_change":"2024-03-29T04:08:53.821709+0000","last_active":"2024-03-29T04:08:54.814624+0000","last_peered":"2024-03-29T04:08:54.814624+0000","last_clean":"2024-03-29T04:08:54.814624+0000","last_became_active":"2024-03-29T04:08:53.820828+0000","last_became_peered":"2024-03-29T04:08:53.820828+0000","last_unstale":"2024-03-29T04:08:54.814624+0000","last_undegraded":"2024-03-29T04:08:54.814624+0000","last_fullsized":"2024-03-29T04:08:54.814624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.821137+0000","last_change":"2024-03-29T04:08:53.820303+0000","last_active":"2024-03-29T04:08:53.821137+0000","last_peered":"2024-03-29T04:08:53.821137+0000","last_clean":"2024-03-29T04:08:53.821137+0000","last_became_active":"2024-03-29T04:08:53.820024+0000","last_became_peered":"2024-03-29T04:08:53.820024+0000","last_unstale":"2024-03-29T04:08:53.821137+0000","last_undegraded":"2024-03-29T04:08:53.821137+0000","last_fullsized":"2024-03-29T04:08:53.821137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815813+0000","last_change":"2024-03-29T04:08:53.824803+0000","last_active":"2024-03-29T04:08:54.815813+0000","last_peered":"2024-03-29T04:08:54.815813+0000","last_clean":"2024-03-29T04:08:54.815813+0000","last_became_active":"2024-03-29T04:08:53.824454+0000","last_became_peered":"2024-03-29T04:08:53.824454+0000","last_unstale":"2024-03-29T04:08:54.815813+0000","last_undegraded":"2024-03-29T04:08:54.815813+0000","last_fullsized":"2024-03-29T04:08:54.815813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813758+0000","last_change":"2024-03-29T04:08:53.819297+0000","last_active":"2024-03-29T04:08:54.813758+0000","last_peered":"2024-03-29T04:08:54.813758+0000","last_clean":"2024-03-29T04:08:54.813758+0000","last_became_active":"2024-03-29T04:08:53.819016+0000","last_became_peered":"2024-03-29T04:08:53.819016+0000","last_unstale":"2024-03-29T04:08:54.813758+0000","last_undegraded":"2024-03-29T04:08:54.813758+0000","last_fullsized":"2024-03-29T04:08:54.813758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.830662+0000","last_change":"2024-03-29T04:08:52.815351+0000","last_active":"2024-03-29T04:08:53.830662+0000","last_peered":"2024-03-29T04:08:53.830662+0000","last_clean":"2024-03-29T04:08:53.830662+0000","last_became_active":"2024-03-29T04:08:52.814942+0000","last_became_peered":"2024-03-29T04:08:52.814942+0000","last_unstale":"2024-03-29T04:08:53.830662+0000","last_undegraded":"2024-03-29T04:08:53.830662+0000","last_fullsized":"2024-03-29T04:08:53.830662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823606+0000","last_change":"2024-03-29T04:08:53.820823+0000","last_active":"2024-03-29T04:08:53.823606+0000","last_peered":"2024-03-29T04:08:53.823606+0000","last_clean":"2024-03-29T04:08:53.823606+0000","last_became_active":"2024-03-29T04:08:53.820429+0000","last_became_peered":"2024-03-29T04:08:53.820429+0000","last_unstale":"2024-03-29T04:08:53.823606+0000","last_undegraded":"2024-03-29T04:08:53.823606+0000","last_fullsized":"2024-03-29T04:08:53.823606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815991+0000","last_change":"2024-03-29T04:08:52.812694+0000","last_active":"2024-03-29T04:08:54.815991+0000","last_peered":"2024-03-29T04:08:54.815991+0000","last_clean":"2024-03-29T04:08:54.815991+0000","last_became_active":"2024-03-29T04:08:52.812429+0000","last_became_peered":"2024-03-29T04:08:52.812429+0000","last_unstale":"2024-03-29T04:08:54.815991+0000","last_undegraded":"2024-03-29T04:08:54.815991+0000","last_fullsized":"2024-03-29T04:08:54.815991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.808199+0000","last_change":"2024-03-29T04:08:52.809190+0000","last_active":"2024-03-29T04:08:53.808199+0000","last_peered":"2024-03-29T04:08:53.808199+0000","last_clean":"2024-03-29T04:08:53.808199+0000","last_became_active":"2024-03-29T04:08:52.808923+0000","last_became_peered":"2024-03-29T04:08:52.808923+0000","last_unstale":"2024-03-29T04:08:53.808199+0000","last_undegraded":"2024-03-29T04:08:53.808199+0000","last_fullsized":"2024-03-29T04:08:53.808199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831108+0000","last_change":"2024-03-29T04:08:53.828865+0000","last_active":"2024-03-29T04:08:53.831108+0000","last_peered":"2024-03-29T04:08:53.831108+0000","last_clean":"2024-03-29T04:08:53.831108+0000","last_became_active":"2024-03-29T04:08:53.828601+0000","last_became_peered":"2024-03-29T04:08:53.828601+0000","last_unstale":"2024-03-29T04:08:53.831108+0000","last_undegraded":"2024-03-29T04:08:53.831108+0000","last_fullsized":"2024-03-29T04:08:53.831108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:23.273277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814227+0000","last_change":"2024-03-29T04:08:52.810336+0000","last_active":"2024-03-29T04:08:54.814227+0000","last_peered":"2024-03-29T04:08:54.814227+0000","last_clean":"2024-03-29T04:08:54.814227+0000","last_became_active":"2024-03-29T04:08:52.810021+0000","last_became_peered":"2024-03-29T04:08:52.810021+0000","last_unstale":"2024-03-29T04:08:54.814227+0000","last_undegraded":"2024-03-29T04:08:54.814227+0000","last_fullsized":"2024-03-29T04:08:54.814227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811960+0000","last_change":"2024-03-29T04:08:53.817702+0000","last_active":"2024-03-29T04:08:54.811960+0000","last_peered":"2024-03-29T04:08:54.811960+0000","last_clean":"2024-03-29T04:08:54.811960+0000","last_became_active":"2024-03-29T04:08:53.817117+0000","last_became_peered":"2024-03-29T04:08:53.817117+0000","last_unstale":"2024-03-29T04:08:54.811960+0000","last_undegraded":"2024-03-29T04:08:54.811960+0000","last_fullsized":"2024-03-29T04:08:54.811960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816011+0000","last_change":"2024-03-29T04:08:52.812898+0000","last_active":"2024-03-29T04:08:54.816011+0000","last_peered":"2024-03-29T04:08:54.816011+0000","last_clean":"2024-03-29T04:08:54.816011+0000","last_became_active":"2024-03-29T04:08:52.811143+0000","last_became_peered":"2024-03-29T04:08:52.811143+0000","last_unstale":"2024-03-29T04:08:54.816011+0000","last_undegraded":"2024-03-29T04:08:54.816011+0000","last_fullsized":"2024-03-29T04:08:54.816011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813759+0000","last_change":"2024-03-29T04:08:53.815139+0000","last_active":"2024-03-29T04:08:54.813759+0000","last_peered":"2024-03-29T04:08:54.813759+0000","last_clean":"2024-03-29T04:08:54.813759+0000","last_became_active":"2024-03-29T04:08:53.814836+0000","last_became_peered":"2024-03-29T04:08:53.814836+0000","last_unstale":"2024-03-29T04:08:54.813759+0000","last_undegraded":"2024-03-29T04:08:54.813759+0000","last_fullsized":"2024-03-29T04:08:54.813759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813822+0000","last_change":"2024-03-29T04:08:52.812383+0000","last_active":"2024-03-29T04:08:54.813822+0000","last_peered":"2024-03-29T04:08:54.813822+0000","last_clean":"2024-03-29T04:08:54.813822+0000","last_became_active":"2024-03-29T04:08:52.810343+0000","last_became_peered":"2024-03-29T04:08:52.810343+0000","last_unstale":"2024-03-29T04:08:54.813822+0000","last_undegraded":"2024-03-29T04:08:54.813822+0000","last_fullsized":"2024-03-29T04:08:54.813822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823577+0000","last_change":"2024-03-29T04:08:53.821898+0000","last_active":"2024-03-29T04:08:53.823577+0000","last_peered":"2024-03-29T04:08:53.823577+0000","last_clean":"2024-03-29T04:08:53.823577+0000","last_became_active":"2024-03-29T04:08:53.821126+0000","last_became_peered":"2024-03-29T04:08:53.821126+0000","last_unstale":"2024-03-29T04:08:53.823577+0000","last_undegraded":"2024-03-29T04:08:53.823577+0000","last_fullsized":"2024-03-29T04:08:53.823577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812079+0000","last_change":"2024-03-29T04:08:52.810835+0000","last_active":"2024-03-29T04:08:54.812079+0000","last_peered":"2024-03-29T04:08:54.812079+0000","last_clean":"2024-03-29T04:08:54.812079+0000","last_became_active":"2024-03-29T04:08:52.810585+0000","last_became_peered":"2024-03-29T04:08:52.810585+0000","last_unstale":"2024-03-29T04:08:54.812079+0000","last_undegraded":"2024-03-29T04:08:54.812079+0000","last_fullsized":"2024-03-29T04:08:54.812079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814427+0000","last_change":"2024-03-29T04:08:53.817718+0000","last_active":"2024-03-29T04:08:54.814427+0000","last_peered":"2024-03-29T04:08:54.814427+0000","last_clean":"2024-03-29T04:08:54.814427+0000","last_became_active":"2024-03-29T04:08:53.817419+0000","last_became_peered":"2024-03-29T04:08:53.817419+0000","last_unstale":"2024-03-29T04:08:54.814427+0000","last_undegraded":"2024-03-29T04:08:54.814427+0000","last_fullsized":"2024-03-29T04:08:54.814427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816294+0000","last_change":"2024-03-29T04:08:52.809428+0000","last_active":"2024-03-29T04:08:54.816294+0000","last_peered":"2024-03-29T04:08:54.816294+0000","last_clean":"2024-03-29T04:08:54.816294+0000","last_became_active":"2024-03-29T04:08:52.809124+0000","last_became_peered":"2024-03-29T04:08:52.809124+0000","last_unstale":"2024-03-29T04:08:54.816294+0000","last_undegraded":"2024-03-29T04:08:54.816294+0000","last_fullsized":"2024-03-29T04:08:54.816294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546068+0000","last_change":"2024-03-29T04:08:53.823456+0000","last_active":"2024-03-29T04:08:55.546068+0000","last_peered":"2024-03-29T04:08:55.546068+0000","last_clean":"2024-03-29T04:08:55.546068+0000","last_became_active":"2024-03-29T04:08:53.823196+0000","last_became_peered":"2024-03-29T04:08:53.823196+0000","last_unstale":"2024-03-29T04:08:55.546068+0000","last_undegraded":"2024-03-29T04:08:55.546068+0000","last_fullsized":"2024-03-29T04:08:55.546068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.828278+0000","last_change":"2024-03-29T04:08:52.804947+0000","last_active":"2024-03-29T04:08:53.828278+0000","last_peered":"2024-03-29T04:08:53.828278+0000","last_clean":"2024-03-29T04:08:53.828278+0000","last_became_active":"2024-03-29T04:08:52.804675+0000","last_became_peered":"2024-03-29T04:08:52.804675+0000","last_unstale":"2024-03-29T04:08:53.828278+0000","last_undegraded":"2024-03-29T04:08:53.828278+0000","last_fullsized":"2024-03-29T04:08:53.828278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823262+0000","last_change":"2024-03-29T04:08:53.815297+0000","last_active":"2024-03-29T04:08:53.823262+0000","last_peered":"2024-03-29T04:08:53.823262+0000","last_clean":"2024-03-29T04:08:53.823262+0000","last_became_active":"2024-03-29T04:08:53.815003+0000","last_became_peered":"2024-03-29T04:08:53.815003+0000","last_unstale":"2024-03-29T04:08:53.823262+0000","last_undegraded":"2024-03-29T04:08:53.823262+0000","last_fullsized":"2024-03-29T04:08:53.823262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:16.901767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814596+0000","last_change":"2024-03-29T04:08:52.809942+0000","last_active":"2024-03-29T04:08:54.814596+0000","last_peered":"2024-03-29T04:08:54.814596+0000","last_clean":"2024-03-29T04:08:54.814596+0000","last_became_active":"2024-03-29T04:08:52.809671+0000","last_became_peered":"2024-03-29T04:08:52.809671+0000","last_unstale":"2024-03-29T04:08:54.814596+0000","last_undegraded":"2024-03-29T04:08:54.814596+0000","last_fullsized":"2024-03-29T04:08:54.814596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546150+0000","last_change":"2024-03-29T04:08:53.818585+0000","last_active":"2024-03-29T04:08:55.546150+0000","last_peered":"2024-03-29T04:08:55.546150+0000","last_clean":"2024-03-29T04:08:55.546150+0000","last_became_active":"2024-03-29T04:08:53.818252+0000","last_became_peered":"2024-03-29T04:08:53.818252+0000","last_unstale":"2024-03-29T04:08:55.546150+0000","last_undegraded":"2024-03-29T04:08:55.546150+0000","last_fullsized":"2024-03-29T04:08:55.546150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815950+0000","last_change":"2024-03-29T04:08:52.810780+0000","last_active":"2024-03-29T04:08:54.815950+0000","last_peered":"2024-03-29T04:08:54.815950+0000","last_clean":"2024-03-29T04:08:54.815950+0000","last_became_active":"2024-03-29T04:08:52.810487+0000","last_became_peered":"2024-03-29T04:08:52.810487+0000","last_unstale":"2024-03-29T04:08:54.815950+0000","last_undegraded":"2024-03-29T04:08:54.815950+0000","last_fullsized":"2024-03-29T04:08:54.815950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816134+0000","last_change":"2024-03-29T04:08:53.825665+0000","last_active":"2024-03-29T04:08:54.816134+0000","last_peered":"2024-03-29T04:08:54.816134+0000","last_clean":"2024-03-29T04:08:54.816134+0000","last_became_active":"2024-03-29T04:08:53.825229+0000","last_became_peered":"2024-03-29T04:08:53.825229+0000","last_unstale":"2024-03-29T04:08:54.816134+0000","last_undegraded":"2024-03-29T04:08:54.816134+0000","last_fullsized":"2024-03-29T04:08:54.816134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.830026+0000","last_change":"2024-03-29T04:08:52.807491+0000","last_active":"2024-03-29T04:08:53.830026+0000","last_peered":"2024-03-29T04:08:53.830026+0000","last_clean":"2024-03-29T04:08:53.830026+0000","last_became_active":"2024-03-29T04:08:52.807236+0000","last_became_peered":"2024-03-29T04:08:52.807236+0000","last_unstale":"2024-03-29T04:08:53.830026+0000","last_undegraded":"2024-03-29T04:08:53.830026+0000","last_fullsized":"2024-03-29T04:08:53.830026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545849+0000","last_change":"2024-03-29T04:08:53.819664+0000","last_active":"2024-03-29T04:08:55.545849+0000","last_peered":"2024-03-29T04:08:55.545849+0000","last_clean":"2024-03-29T04:08:55.545849+0000","last_became_active":"2024-03-29T04:08:53.819414+0000","last_became_peered":"2024-03-29T04:08:53.819414+0000","last_unstale":"2024-03-29T04:08:55.545849+0000","last_undegraded":"2024-03-29T04:08:55.545849+0000","last_fullsized":"2024-03-29T04:08:55.545849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815063+0000","last_change":"2024-03-29T04:08:52.806445+0000","last_active":"2024-03-29T04:08:54.815063+0000","last_peered":"2024-03-29T04:08:54.815063+0000","last_clean":"2024-03-29T04:08:54.815063+0000","last_became_active":"2024-03-29T04:08:52.806166+0000","last_became_peered":"2024-03-29T04:08:52.806166+0000","last_unstale":"2024-03-29T04:08:54.815063+0000","last_undegraded":"2024-03-29T04:08:54.815063+0000","last_fullsized":"2024-03-29T04:08:54.815063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:03.546395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546168+0000","last_change":"2024-03-29T04:08:53.819783+0000","last_active":"2024-03-29T04:08:55.546168+0000","last_peered":"2024-03-29T04:08:55.546168+0000","last_clean":"2024-03-29T04:08:55.546168+0000","last_became_active":"2024-03-29T04:08:53.819507+0000","last_became_peered":"2024-03-29T04:08:53.819507+0000","last_unstale":"2024-03-29T04:08:55.546168+0000","last_undegraded":"2024-03-29T04:08:55.546168+0000","last_fullsized":"2024-03-29T04:08:55.546168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831487+0000","last_change":"2024-03-29T04:08:52.812349+0000","last_active":"2024-03-29T04:08:53.831487+0000","last_peered":"2024-03-29T04:08:53.831487+0000","last_clean":"2024-03-29T04:08:53.831487+0000","last_became_active":"2024-03-29T04:08:52.812081+0000","last_became_peered":"2024-03-29T04:08:52.812081+0000","last_unstale":"2024-03-29T04:08:53.831487+0000","last_undegraded":"2024-03-29T04:08:53.831487+0000","last_fullsized":"2024-03-29T04:08:53.831487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546387+0000","last_change":"2024-03-29T04:08:53.827351+0000","last_active":"2024-03-29T04:08:55.546387+0000","last_peered":"2024-03-29T04:08:55.546387+0000","last_clean":"2024-03-29T04:08:55.546387+0000","last_became_active":"2024-03-29T04:08:53.826255+0000","last_became_peered":"2024-03-29T04:08:53.826255+0000","last_unstale":"2024-03-29T04:08:55.546387+0000","last_undegraded":"2024-03-29T04:08:55.546387+0000","last_fullsized":"2024-03-29T04:08:55.546387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812164+0000","last_change":"2024-03-29T04:08:52.805752+0000","last_active":"2024-03-29T04:08:54.812164+0000","last_peered":"2024-03-29T04:08:54.812164+0000","last_clean":"2024-03-29T04:08:54.812164+0000","last_became_active":"2024-03-29T04:08:52.804921+0000","last_became_peered":"2024-03-29T04:08:52.804921+0000","last_unstale":"2024-03-29T04:08:54.812164+0000","last_undegraded":"2024-03-29T04:08:54.812164+0000","last_fullsized":"2024-03-29T04:08:54.812164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815243+0000","last_change":"2024-03-29T04:08:53.821399+0000","last_active":"2024-03-29T04:08:54.815243+0000","last_peered":"2024-03-29T04:08:54.815243+0000","last_clean":"2024-03-29T04:08:54.815243+0000","last_became_active":"2024-03-29T04:08:53.821026+0000","last_became_peered":"2024-03-29T04:08:53.821026+0000","last_unstale":"2024-03-29T04:08:54.815243+0000","last_undegraded":"2024-03-29T04:08:54.815243+0000","last_fullsized":"2024-03-29T04:08:54.815243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.809520+0000","last_change":"2024-03-29T04:08:52.811397+0000","last_active":"2024-03-29T04:08:53.809520+0000","last_peered":"2024-03-29T04:08:53.809520+0000","last_clean":"2024-03-29T04:08:53.809520+0000","last_became_active":"2024-03-29T04:08:52.810968+0000","last_became_peered":"2024-03-29T04:08:52.810968+0000","last_unstale":"2024-03-29T04:08:53.809520+0000","last_undegraded":"2024-03-29T04:08:53.809520+0000","last_fullsized":"2024-03-29T04:08:53.809520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546224+0000","last_change":"2024-03-29T04:08:53.824444+0000","last_active":"2024-03-29T04:08:55.546224+0000","last_peered":"2024-03-29T04:08:55.546224+0000","last_clean":"2024-03-29T04:08:55.546224+0000","last_became_active":"2024-03-29T04:08:53.821578+0000","last_became_peered":"2024-03-29T04:08:53.821578+0000","last_unstale":"2024-03-29T04:08:55.546224+0000","last_undegraded":"2024-03-29T04:08:55.546224+0000","last_fullsized":"2024-03-29T04:08:55.546224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545812+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:08:55.545812+0000","last_peered":"2024-03-29T04:08:55.545812+0000","last_clean":"2024-03-29T04:08:55.545812+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:08:55.545812+0000","last_undegraded":"2024-03-29T04:08:55.545812+0000","last_fullsized":"2024-03-29T04:08:55.545812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813649+0000","last_change":"2024-03-29T04:08:52.807738+0000","last_active":"2024-03-29T04:08:54.813649+0000","last_peered":"2024-03-29T04:08:54.813649+0000","last_clean":"2024-03-29T04:08:54.813649+0000","last_became_active":"2024-03-29T04:08:52.807442+0000","last_became_peered":"2024-03-29T04:08:52.807442+0000","last_unstale":"2024-03-29T04:08:54.813649+0000","last_undegraded":"2024-03-29T04:08:54.813649+0000","last_fullsized":"2024-03-29T04:08:54.813649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.826424+0000","last_change":"2024-03-29T04:08:53.825630+0000","last_active":"2024-03-29T04:08:53.826424+0000","last_peered":"2024-03-29T04:08:53.826424+0000","last_clean":"2024-03-29T04:08:53.826424+0000","last_became_active":"2024-03-29T04:08:53.825386+0000","last_became_peered":"2024-03-29T04:08:53.825386+0000","last_unstale":"2024-03-29T04:08:53.826424+0000","last_undegraded":"2024-03-29T04:08:53.826424+0000","last_fullsized":"2024-03-29T04:08:53.826424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545708+0000","last_change":"2024-03-29T04:08:52.820187+0000","last_active":"2024-03-29T04:08:55.545708+0000","last_peered":"2024-03-29T04:08:55.545708+0000","last_clean":"2024-03-29T04:08:55.545708+0000","last_became_active":"2024-03-29T04:08:52.819912+0000","last_became_peered":"2024-03-29T04:08:52.819912+0000","last_unstale":"2024-03-29T04:08:55.545708+0000","last_undegraded":"2024-03-29T04:08:55.545708+0000","last_fullsized":"2024-03-29T04:08:55.545708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823307+0000","last_change":"2024-03-29T04:08:53.815607+0000","last_active":"2024-03-29T04:08:53.823307+0000","last_peered":"2024-03-29T04:08:53.823307+0000","last_clean":"2024-03-29T04:08:53.823307+0000","last_became_active":"2024-03-29T04:08:53.814274+0000","last_became_peered":"2024-03-29T04:08:53.814274+0000","last_unstale":"2024-03-29T04:08:53.823307+0000","last_undegraded":"2024-03-29T04:08:53.823307+0000","last_fullsized":"2024-03-29T04:08:53.823307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:18.995986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.809640+0000","last_change":"2024-03-29T04:08:52.813143+0000","last_active":"2024-03-29T04:08:53.809640+0000","last_peered":"2024-03-29T04:08:53.809640+0000","last_clean":"2024-03-29T04:08:53.809640+0000","last_became_active":"2024-03-29T04:08:52.812866+0000","last_became_peered":"2024-03-29T04:08:52.812866+0000","last_unstale":"2024-03-29T04:08:53.809640+0000","last_undegraded":"2024-03-29T04:08:53.809640+0000","last_fullsized":"2024-03-29T04:08:53.809640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811826+0000","last_change":"2024-03-29T04:08:53.818488+0000","last_active":"2024-03-29T04:08:54.811826+0000","last_peered":"2024-03-29T04:08:54.811826+0000","last_clean":"2024-03-29T04:08:54.811826+0000","last_became_active":"2024-03-29T04:08:53.818222+0000","last_became_peered":"2024-03-29T04:08:53.818222+0000","last_unstale":"2024-03-29T04:08:54.811826+0000","last_undegraded":"2024-03-29T04:08:54.811826+0000","last_fullsized":"2024-03-29T04:08:54.811826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814714+0000","last_change":"2024-03-29T04:08:52.808830+0000","last_active":"2024-03-29T04:08:54.814714+0000","last_peered":"2024-03-29T04:08:54.814714+0000","last_clean":"2024-03-29T04:08:54.814714+0000","last_became_active":"2024-03-29T04:08:52.808524+0000","last_became_peered":"2024-03-29T04:08:52.808524+0000","last_unstale":"2024-03-29T04:08:54.814714+0000","last_undegraded":"2024-03-29T04:08:54.814714+0000","last_fullsized":"2024-03-29T04:08:54.814714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815259+0000","last_change":"2024-03-29T04:08:53.823062+0000","last_active":"2024-03-29T04:08:54.815259+0000","last_peered":"2024-03-29T04:08:54.815259+0000","last_clean":"2024-03-29T04:08:54.815259+0000","last_became_active":"2024-03-29T04:08:53.822666+0000","last_became_peered":"2024-03-29T04:08:53.822666+0000","last_unstale":"2024-03-29T04:08:54.815259+0000","last_undegraded":"2024-03-29T04:08:54.815259+0000","last_fullsized":"2024-03-29T04:08:54.815259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812019+0000","last_change":"2024-03-29T04:08:52.799226+0000","last_active":"2024-03-29T04:08:54.812019+0000","last_peered":"2024-03-29T04:08:54.812019+0000","last_clean":"2024-03-29T04:08:54.812019+0000","last_became_active":"2024-03-29T04:08:52.798964+0000","last_became_peered":"2024-03-29T04:08:52.798964+0000","last_unstale":"2024-03-29T04:08:54.812019+0000","last_undegraded":"2024-03-29T04:08:54.812019+0000","last_fullsized":"2024-03-29T04:08:54.812019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815451+0000","last_change":"2024-03-29T04:08:53.821737+0000","last_active":"2024-03-29T04:08:54.815451+0000","last_peered":"2024-03-29T04:08:54.815451+0000","last_clean":"2024-03-29T04:08:54.815451+0000","last_became_active":"2024-03-29T04:08:53.820680+0000","last_became_peered":"2024-03-29T04:08:53.820680+0000","last_unstale":"2024-03-29T04:08:54.815451+0000","last_undegraded":"2024-03-29T04:08:54.815451+0000","last_fullsized":"2024-03-29T04:08:54.815451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546333+0000","last_change":"2024-03-29T04:08:52.821118+0000","last_active":"2024-03-29T04:08:55.546333+0000","last_peered":"2024-03-29T04:08:55.546333+0000","last_clean":"2024-03-29T04:08:55.546333+0000","last_became_active":"2024-03-29T04:08:52.820549+0000","last_became_peered":"2024-03-29T04:08:52.820549+0000","last_unstale":"2024-03-29T04:08:55.546333+0000","last_undegraded":"2024-03-29T04:08:55.546333+0000","last_fullsized":"2024-03-29T04:08:55.546333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811978+0000","last_change":"2024-03-29T04:08:53.810781+0000","last_active":"2024-03-29T04:08:54.811978+0000","last_peered":"2024-03-29T04:08:54.811978+0000","last_clean":"2024-03-29T04:08:54.811978+0000","last_became_active":"2024-03-29T04:08:53.810518+0000","last_became_peered":"2024-03-29T04:08:53.810518+0000","last_unstale":"2024-03-29T04:08:54.811978+0000","last_undegraded":"2024-03-29T04:08:54.811978+0000","last_fullsized":"2024-03-29T04:08:54.811978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816447+0000","last_change":"2024-03-29T04:08:52.809876+0000","last_active":"2024-03-29T04:08:54.816447+0000","last_peered":"2024-03-29T04:08:54.816447+0000","last_clean":"2024-03-29T04:08:54.816447+0000","last_became_active":"2024-03-29T04:08:52.809575+0000","last_became_peered":"2024-03-29T04:08:52.809575+0000","last_unstale":"2024-03-29T04:08:54.816447+0000","last_undegraded":"2024-03-29T04:08:54.816447+0000","last_fullsized":"2024-03-29T04:08:54.816447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.832275+0000","last_change":"2024-03-29T04:08:53.831225+0000","last_active":"2024-03-29T04:08:53.832275+0000","last_peered":"2024-03-29T04:08:53.832275+0000","last_clean":"2024-03-29T04:08:53.832275+0000","last_became_active":"2024-03-29T04:08:53.830819+0000","last_became_peered":"2024-03-29T04:08:53.830819+0000","last_unstale":"2024-03-29T04:08:53.832275+0000","last_undegraded":"2024-03-29T04:08:53.832275+0000","last_fullsized":"2024-03-29T04:08:53.832275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816132+0000","last_change":"2024-03-29T04:08:52.812725+0000","last_active":"2024-03-29T04:08:54.816132+0000","last_peered":"2024-03-29T04:08:54.816132+0000","last_clean":"2024-03-29T04:08:54.816132+0000","last_became_active":"2024-03-29T04:08:52.812020+0000","last_became_peered":"2024-03-29T04:08:52.812020+0000","last_unstale":"2024-03-29T04:08:54.816132+0000","last_undegraded":"2024-03-29T04:08:54.816132+0000","last_fullsized":"2024-03-29T04:08:54.816132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.816359+0000","last_change":"2024-03-29T04:08:53.811809+0000","last_active":"2024-03-29T04:08:53.816359+0000","last_peered":"2024-03-29T04:08:53.816359+0000","last_clean":"2024-03-29T04:08:53.816359+0000","last_became_active":"2024-03-29T04:08:53.811513+0000","last_became_peered":"2024-03-29T04:08:53.811513+0000","last_unstale":"2024-03-29T04:08:53.816359+0000","last_undegraded":"2024-03-29T04:08:53.816359+0000","last_fullsized":"2024-03-29T04:08:53.816359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:59.816889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545699+0000","last_change":"2024-03-29T04:08:52.814972+0000","last_active":"2024-03-29T04:08:55.545699+0000","last_peered":"2024-03-29T04:08:55.545699+0000","last_clean":"2024-03-29T04:08:55.545699+0000","last_became_active":"2024-03-29T04:08:52.814714+0000","last_became_peered":"2024-03-29T04:08:52.814714+0000","last_unstale":"2024-03-29T04:08:55.545699+0000","last_undegraded":"2024-03-29T04:08:55.545699+0000","last_fullsized":"2024-03-29T04:08:55.545699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814365+0000","last_change":"2024-03-29T04:08:53.817677+0000","last_active":"2024-03-29T04:08:54.814365+0000","last_peered":"2024-03-29T04:08:54.814365+0000","last_clean":"2024-03-29T04:08:54.814365+0000","last_became_active":"2024-03-29T04:08:53.815481+0000","last_became_peered":"2024-03-29T04:08:53.815481+0000","last_unstale":"2024-03-29T04:08:54.814365+0000","last_undegraded":"2024-03-29T04:08:54.814365+0000","last_fullsized":"2024-03-29T04:08:54.814365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811927+0000","last_change":"2024-03-29T04:08:52.814031+0000","last_active":"2024-03-29T04:08:54.811927+0000","last_peered":"2024-03-29T04:08:54.811927+0000","last_clean":"2024-03-29T04:08:54.811927+0000","last_became_active":"2024-03-29T04:08:52.813714+0000","last_became_peered":"2024-03-29T04:08:52.813714+0000","last_unstale":"2024-03-29T04:08:54.811927+0000","last_undegraded":"2024-03-29T04:08:54.811927+0000","last_fullsized":"2024-03-29T04:08:54.811927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823768+0000","last_change":"2024-03-29T04:08:53.818653+0000","last_active":"2024-03-29T04:08:53.823768+0000","last_peered":"2024-03-29T04:08:53.823768+0000","last_clean":"2024-03-29T04:08:53.823768+0000","last_became_active":"2024-03-29T04:08:53.818387+0000","last_became_peered":"2024-03-29T04:08:53.818387+0000","last_unstale":"2024-03-29T04:08:53.823768+0000","last_undegraded":"2024-03-29T04:08:53.823768+0000","last_fullsized":"2024-03-29T04:08:53.823768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812062+0000","last_change":"2024-03-29T04:08:52.806456+0000","last_active":"2024-03-29T04:08:54.812062+0000","last_peered":"2024-03-29T04:08:54.812062+0000","last_clean":"2024-03-29T04:08:54.812062+0000","last_became_active":"2024-03-29T04:08:52.806184+0000","last_became_peered":"2024-03-29T04:08:52.806184+0000","last_unstale":"2024-03-29T04:08:54.812062+0000","last_undegraded":"2024-03-29T04:08:54.812062+0000","last_fullsized":"2024-03-29T04:08:54.812062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831092+0000","last_change":"2024-03-29T04:08:53.828867+0000","last_active":"2024-03-29T04:08:53.831092+0000","last_peered":"2024-03-29T04:08:53.831092+0000","last_clean":"2024-03-29T04:08:53.831092+0000","last_became_active":"2024-03-29T04:08:53.828610+0000","last_became_peered":"2024-03-29T04:08:53.828610+0000","last_unstale":"2024-03-29T04:08:53.831092+0000","last_undegraded":"2024-03-29T04:08:53.831092+0000","last_fullsized":"2024-03-29T04:08:53.831092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546226+0000","last_change":"2024-03-29T04:08:52.821021+0000","last_active":"2024-03-29T04:08:55.546226+0000","last_peered":"2024-03-29T04:08:55.546226+0000","last_clean":"2024-03-29T04:08:55.546226+0000","last_became_active":"2024-03-29T04:08:52.820478+0000","last_became_peered":"2024-03-29T04:08:52.820478+0000","last_unstale":"2024-03-29T04:08:55.546226+0000","last_undegraded":"2024-03-29T04:08:55.546226+0000","last_fullsized":"2024-03-29T04:08:55.546226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815868+0000","last_change":"2024-03-29T04:08:53.825032+0000","last_active":"2024-03-29T04:08:54.815868+0000","last_peered":"2024-03-29T04:08:54.815868+0000","last_clean":"2024-03-29T04:08:54.815868+0000","last_became_active":"2024-03-29T04:08:53.824740+0000","last_became_peered":"2024-03-29T04:08:53.824740+0000","last_unstale":"2024-03-29T04:08:54.815868+0000","last_undegraded":"2024-03-29T04:08:54.815868+0000","last_fullsized":"2024-03-29T04:08:54.815868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815425+0000","last_change":"2024-03-29T04:08:53.816669+0000","last_active":"2024-03-29T04:08:54.815425+0000","last_peered":"2024-03-29T04:08:54.815425+0000","last_clean":"2024-03-29T04:08:54.815425+0000","last_became_active":"2024-03-29T04:08:53.816348+0000","last_became_peered":"2024-03-29T04:08:53.816348+0000","last_unstale":"2024-03-29T04:08:54.815425+0000","last_undegraded":"2024-03-29T04:08:54.815425+0000","last_fullsized":"2024-03-29T04:08:54.815425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545977+0000","last_change":"2024-03-29T04:08:52.820909+0000","last_active":"2024-03-29T04:08:55.545977+0000","last_peered":"2024-03-29T04:08:55.545977+0000","last_clean":"2024-03-29T04:08:55.545977+0000","last_became_active":"2024-03-29T04:08:52.820523+0000","last_became_peered":"2024-03-29T04:08:52.820523+0000","last_unstale":"2024-03-29T04:08:55.545977+0000","last_undegraded":"2024-03-29T04:08:55.545977+0000","last_fullsized":"2024-03-29T04:08:55.545977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815127+0000","last_change":"2024-03-29T04:08:53.817769+0000","last_active":"2024-03-29T04:08:54.815127+0000","last_peered":"2024-03-29T04:08:54.815127+0000","last_clean":"2024-03-29T04:08:54.815127+0000","last_became_active":"2024-03-29T04:08:53.816753+0000","last_became_peered":"2024-03-29T04:08:53.816753+0000","last_unstale":"2024-03-29T04:08:54.815127+0000","last_undegraded":"2024-03-29T04:08:54.815127+0000","last_fullsized":"2024-03-29T04:08:54.815127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546206+0000","last_change":"2024-03-29T04:08:52.820167+0000","last_active":"2024-03-29T04:08:55.546206+0000","last_peered":"2024-03-29T04:08:55.546206+0000","last_clean":"2024-03-29T04:08:55.546206+0000","last_became_active":"2024-03-29T04:08:52.819858+0000","last_became_peered":"2024-03-29T04:08:52.819858+0000","last_unstale":"2024-03-29T04:08:55.546206+0000","last_undegraded":"2024-03-29T04:08:55.546206+0000","last_fullsized":"2024-03-29T04:08:55.546206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823596+0000","last_change":"2024-03-29T04:08:53.822001+0000","last_active":"2024-03-29T04:08:53.823596+0000","last_peered":"2024-03-29T04:08:53.823596+0000","last_clean":"2024-03-29T04:08:53.823596+0000","last_became_active":"2024-03-29T04:08:53.821669+0000","last_became_peered":"2024-03-29T04:08:53.821669+0000","last_unstale":"2024-03-29T04:08:53.823596+0000","last_undegraded":"2024-03-29T04:08:53.823596+0000","last_fullsized":"2024-03-29T04:08:53.823596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:23.273277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.830974+0000","last_change":"2024-03-29T04:08:52.807347+0000","last_active":"2024-03-29T04:08:53.830974+0000","last_peered":"2024-03-29T04:08:53.830974+0000","last_clean":"2024-03-29T04:08:53.830974+0000","last_became_active":"2024-03-29T04:08:52.807042+0000","last_became_peered":"2024-03-29T04:08:52.807042+0000","last_unstale":"2024-03-29T04:08:53.830974+0000","last_undegraded":"2024-03-29T04:08:53.830974+0000","last_fullsized":"2024-03-29T04:08:53.830974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831375+0000","last_change":"2024-03-29T04:08:53.829560+0000","last_active":"2024-03-29T04:08:53.831375+0000","last_peered":"2024-03-29T04:08:53.831375+0000","last_clean":"2024-03-29T04:08:53.831375+0000","last_became_active":"2024-03-29T04:08:53.829312+0000","last_became_peered":"2024-03-29T04:08:53.829312+0000","last_unstale":"2024-03-29T04:08:53.831375+0000","last_undegraded":"2024-03-29T04:08:53.831375+0000","last_fullsized":"2024-03-29T04:08:53.831375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:39.294386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.829143+0000","last_change":"2024-03-29T04:08:52.813322+0000","last_active":"2024-03-29T04:08:53.829143+0000","last_peered":"2024-03-29T04:08:53.829143+0000","last_clean":"2024-03-29T04:08:53.829143+0000","last_became_active":"2024-03-29T04:08:52.813020+0000","last_became_peered":"2024-03-29T04:08:52.813020+0000","last_unstale":"2024-03-29T04:08:53.829143+0000","last_undegraded":"2024-03-29T04:08:53.829143+0000","last_fullsized":"2024-03-29T04:08:53.829143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":665435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":212992,"data_stored":69797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":68185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":208896,"data_stored":68345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":659191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:08:57.899 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:08:58.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.058+0000 7f219e959640 1 Processor -- start 2024-03-29T04:08:58.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.058+0000 7f219e959640 1 -- start start 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.059+0000 7f219e959640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f219815f750 0x7f21980afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.059+0000 7f219e959640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21980b55f0 con 0x7f219815b450 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.059+0000 7f219e959640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21980af2c0 con 0x7f21980b3580 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.059+0000 7f219e959640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f21980b39c0 con 0x7f219815f750 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.059+0000 7f2197fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f219815b450 0x7f219806e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58962/0 (socket says 172.21.15.156:58962) 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.059+0000 7f2197fff640 1 -- 172.21.15.156:0/3839787744 learned_addr learned my addr 172.21.15.156:0/3839787744 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f219cef6640 1 --2- 172.21.15.156:0/3839787744 >> v2:172.21.15.203:3300/0 conn(0x7f219815f750 0x7f21980afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f2196ffd640 1 -- 172.21.15.156:0/3839787744 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937952033 0 0) 0x7f21980b55f0 con 0x7f219815b450 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f2196ffd640 1 -- 172.21.15.156:0/3839787744 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2180005730 con 0x7f219815b450 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f219cef6640 1 -- 172.21.15.156:0/3839787744 >> v1:172.21.15.203:6789/0 conn(0x7f21980b3580 legacy=0x7f219815f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f2196ffd640 1 -- 172.21.15.156:0/3839787744 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3787339272 0 0) 0x7f2180005730 con 0x7f219815b450 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f219cef6640 1 -- 172.21.15.156:0/3839787744 >> v1:172.21.15.156:6789/0 conn(0x7f219815b450 legacy=0x7f219806e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.060+0000 7f219cef6640 1 -- 172.21.15.156:0/3839787744 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21980af5a0 con 0x7f219815f750 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.061+0000 7f219cef6640 1 --2- 172.21.15.156:0/3839787744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21980afda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2190009d90 tx=0x7f219002bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ff8ddaf37615617 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.061+0000 7f2196ffd640 1 -- 172.21.15.156:0/3839787744 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190039070 con 0x7f219815f750 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.061+0000 7f2196ffd640 1 -- 172.21.15.156:0/3839787744 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219002c830 con 0x7f219815f750 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.061+0000 7f2196ffd640 1 -- 172.21.15.156:0/3839787744 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188001670 con 0x7f219815f750 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.061+0000 7f219e959640 1 -- 172.21.15.156:0/3839787744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 msgr2=0x7f21980afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.061+0000 7f219e959640 1 --2- 172.21.15.156:0/3839787744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21980afda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2190009d90 tx=0x7f219002bd00 comp rx=0 tx=0).stop 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.062+0000 7f219e959640 1 -- 172.21.15.156:0/3839787744 shutdown_connections 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.062+0000 7f219e959640 1 --2- 172.21.15.156:0/3839787744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21980afda0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.062+0000 7f219e959640 1 -- 172.21.15.156:0/3839787744 >> 172.21.15.156:0/3839787744 conn(0x7f21980b68b0 msgr2=0x7f21980b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:58.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.062+0000 7f219e959640 1 -- 172.21.15.156:0/3839787744 shutdown_connections 2024-03-29T04:08:58.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.062+0000 7f219e959640 1 -- 172.21.15.156:0/3839787744 wait complete. 2024-03-29T04:08:58.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.063+0000 7f219e959640 1 Processor -- start 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.063+0000 7f219e959640 1 -- start start 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 0x7f219815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f219815b450 0x7f21980cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21981f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f219815b830 con 0x7f21980b3580 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f219815fd30 con 0x7f219815f750 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2198155cb0 con 0x7f219815b450 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 0x7f219815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 0x7f219815e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45320/0 (socket says 172.21.15.156:45320) 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 -- 172.21.15.156:0/4148998182 learned_addr learned my addr 172.21.15.156:0/4148998182 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219cef6640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21981f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f21977fe640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f219815b450 0x7f21980cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 -- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f219815b450 msgr2=0x7f21980cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f219815b450 0x7f21980cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 -- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 msgr2=0x7f21981f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21981f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 -- 172.21.15.156:0/4148998182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2198155f90 con 0x7f21980b3580 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2197fff640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 0x7f219815e070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f218800c7b0 tx=0x7f218800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188018070 con 0x7f21980b3580 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21880072d0 con 0x7f21980b3580 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2198156270 con 0x7f21980b3580 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21880047e0 con 0x7f21980b3580 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.064+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219815e5b0 con 0x7f21980b3580 2024-03-29T04:08:58.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.065+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2198068d20 con 0x7f21980b3580 2024-03-29T04:08:58.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.066+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f218801b070 con 0x7f21980b3580 2024-03-29T04:08:58.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.067+0000 7f2194ff9640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f218406c530 0x7f218406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.067+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f218808ab70 con 0x7f21980b3580 2024-03-29T04:08:58.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.067+0000 7f21977fe640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f218406c530 0x7f218406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.067+0000 7f21977fe640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f218406c530 0x7f218406e9f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f218c000f60 tx=0x7f218c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.071+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2188059190 con 0x7f21980b3580 2024-03-29T04:08:58.175 INFO:tasks.ceph.mds.c.smithi156.stderr:2024-03-29T04:08:58.196+0000 7f92ed70f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T04:08:58.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.248+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21981520a0 con 0x7f218406c530 2024-03-29T04:08:58.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.254+0000 7f2194ff9640 1 -- 172.21.15.156:0/4148998182 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251933 (secure 0 0 0) 0x7f21981520a0 con 0x7f218406c530 2024-03-29T04:08:58.236 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:58.238 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:08:58.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f218406c530 msgr2=0x7f218406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f218406c530 0x7f218406e9f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f218c000f60 tx=0x7f218c009040 comp rx=0 tx=0).stop 2024-03-29T04:08:58.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 msgr2=0x7f219815e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 0x7f219815e070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f218800c7b0 tx=0x7f218800cc70 comp rx=0 tx=0).stop 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 shutdown_connections 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f218406c530 0x7f218406e9f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219815f750 0x7f21981f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f219815b450 0x7f21980cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 --2- 172.21.15.156:0/4148998182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21980b3580 0x7f219815e070 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 >> 172.21.15.156:0/4148998182 conn(0x7f21980b68b0 msgr2=0x7f2198070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 shutdown_connections 2024-03-29T04:08:58.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.258+0000 7f219e959640 1 -- 172.21.15.156:0/4148998182 wait complete. 2024-03-29T04:08:58.248 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-29T04:08:57.637924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235664,"kb_used_data":2840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739056,"statfs":{"total":773094113280,"available":772852793344,"internally_reserved":0,"allocated":2908160,"data_stored":1756236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.848325"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546238+0000","last_change":"2024-03-29T04:08:53.827540+0000","last_active":"2024-03-29T04:08:55.546238+0000","last_peered":"2024-03-29T04:08:55.546238+0000","last_clean":"2024-03-29T04:08:55.546238+0000","last_became_active":"2024-03-29T04:08:53.827301+0000","last_became_peered":"2024-03-29T04:08:53.827301+0000","last_unstale":"2024-03-29T04:08:55.546238+0000","last_undegraded":"2024-03-29T04:08:55.546238+0000","last_fullsized":"2024-03-29T04:08:55.546238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812222+0000","last_change":"2024-03-29T04:08:53.809513+0000","last_active":"2024-03-29T04:08:54.812222+0000","last_peered":"2024-03-29T04:08:54.812222+0000","last_clean":"2024-03-29T04:08:54.812222+0000","last_became_active":"2024-03-29T04:08:53.809236+0000","last_became_peered":"2024-03-29T04:08:53.809236+0000","last_unstale":"2024-03-29T04:08:54.812222+0000","last_undegraded":"2024-03-29T04:08:54.812222+0000","last_fullsized":"2024-03-29T04:08:54.812222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823581+0000","last_change":"2024-03-29T04:08:53.821233+0000","last_active":"2024-03-29T04:08:53.823581+0000","last_peered":"2024-03-29T04:08:53.823581+0000","last_clean":"2024-03-29T04:08:53.823581+0000","last_became_active":"2024-03-29T04:08:53.820557+0000","last_became_peered":"2024-03-29T04:08:53.820557+0000","last_unstale":"2024-03-29T04:08:53.823581+0000","last_undegraded":"2024-03-29T04:08:53.823581+0000","last_fullsized":"2024-03-29T04:08:53.823581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814954+0000","last_change":"2024-03-29T04:08:53.824851+0000","last_active":"2024-03-29T04:08:54.814954+0000","last_peered":"2024-03-29T04:08:54.814954+0000","last_clean":"2024-03-29T04:08:54.814954+0000","last_became_active":"2024-03-29T04:08:53.824563+0000","last_became_peered":"2024-03-29T04:08:53.824563+0000","last_unstale":"2024-03-29T04:08:54.814954+0000","last_undegraded":"2024-03-29T04:08:54.814954+0000","last_fullsized":"2024-03-29T04:08:54.814954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546094+0000","last_change":"2024-03-29T04:08:53.820567+0000","last_active":"2024-03-29T04:08:55.546094+0000","last_peered":"2024-03-29T04:08:55.546094+0000","last_clean":"2024-03-29T04:08:55.546094+0000","last_became_active":"2024-03-29T04:08:53.820290+0000","last_became_peered":"2024-03-29T04:08:53.820290+0000","last_unstale":"2024-03-29T04:08:55.546094+0000","last_undegraded":"2024-03-29T04:08:55.546094+0000","last_fullsized":"2024-03-29T04:08:55.546094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814434+0000","last_change":"2024-03-29T04:08:53.818966+0000","last_active":"2024-03-29T04:08:54.814434+0000","last_peered":"2024-03-29T04:08:54.814434+0000","last_clean":"2024-03-29T04:08:54.814434+0000","last_became_active":"2024-03-29T04:08:53.818396+0000","last_became_peered":"2024-03-29T04:08:53.818396+0000","last_unstale":"2024-03-29T04:08:54.814434+0000","last_undegraded":"2024-03-29T04:08:54.814434+0000","last_fullsized":"2024-03-29T04:08:54.814434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545835+0000","last_change":"2024-03-29T04:08:53.817600+0000","last_active":"2024-03-29T04:08:55.545835+0000","last_peered":"2024-03-29T04:08:55.545835+0000","last_clean":"2024-03-29T04:08:55.545835+0000","last_became_active":"2024-03-29T04:08:53.817308+0000","last_became_peered":"2024-03-29T04:08:53.817308+0000","last_unstale":"2024-03-29T04:08:55.545835+0000","last_undegraded":"2024-03-29T04:08:55.545835+0000","last_fullsized":"2024-03-29T04:08:55.545835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.827597+0000","last_change":"2024-03-29T04:08:53.826247+0000","last_active":"2024-03-29T04:08:53.827597+0000","last_peered":"2024-03-29T04:08:53.827597+0000","last_clean":"2024-03-29T04:08:53.827597+0000","last_became_active":"2024-03-29T04:08:53.825974+0000","last_became_peered":"2024-03-29T04:08:53.825974+0000","last_unstale":"2024-03-29T04:08:53.827597+0000","last_undegraded":"2024-03-29T04:08:53.827597+0000","last_fullsized":"2024-03-29T04:08:53.827597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546330+0000","last_change":"2024-03-29T04:08:53.824041+0000","last_active":"2024-03-29T04:08:55.546330+0000","last_peered":"2024-03-29T04:08:55.546330+0000","last_clean":"2024-03-29T04:08:55.546330+0000","last_became_active":"2024-03-29T04:08:53.823788+0000","last_became_peered":"2024-03-29T04:08:53.823788+0000","last_unstale":"2024-03-29T04:08:55.546330+0000","last_undegraded":"2024-03-29T04:08:55.546330+0000","last_fullsized":"2024-03-29T04:08:55.546330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816424+0000","last_change":"2024-03-29T04:08:53.830205+0000","last_active":"2024-03-29T04:08:54.816424+0000","last_peered":"2024-03-29T04:08:54.816424+0000","last_clean":"2024-03-29T04:08:54.816424+0000","last_became_active":"2024-03-29T04:08:53.829931+0000","last_became_peered":"2024-03-29T04:08:53.829931+0000","last_unstale":"2024-03-29T04:08:54.816424+0000","last_undegraded":"2024-03-29T04:08:54.816424+0000","last_fullsized":"2024-03-29T04:08:54.816424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816205+0000","last_change":"2024-03-29T04:08:53.821458+0000","last_active":"2024-03-29T04:08:54.816205+0000","last_peered":"2024-03-29T04:08:54.816205+0000","last_clean":"2024-03-29T04:08:54.816205+0000","last_became_active":"2024-03-29T04:08:53.821168+0000","last_became_peered":"2024-03-29T04:08:53.821168+0000","last_unstale":"2024-03-29T04:08:54.816205+0000","last_undegraded":"2024-03-29T04:08:54.816205+0000","last_fullsized":"2024-03-29T04:08:54.816205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814279+0000","last_change":"2024-03-29T04:08:53.827058+0000","last_active":"2024-03-29T04:08:54.814279+0000","last_peered":"2024-03-29T04:08:54.814279+0000","last_clean":"2024-03-29T04:08:54.814279+0000","last_became_active":"2024-03-29T04:08:53.826167+0000","last_became_peered":"2024-03-29T04:08:53.826167+0000","last_unstale":"2024-03-29T04:08:54.814279+0000","last_undegraded":"2024-03-29T04:08:54.814279+0000","last_fullsized":"2024-03-29T04:08:54.814279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831543+0000","last_change":"2024-03-29T04:08:53.830239+0000","last_active":"2024-03-29T04:08:53.831543+0000","last_peered":"2024-03-29T04:08:53.831543+0000","last_clean":"2024-03-29T04:08:53.831543+0000","last_became_active":"2024-03-29T04:08:53.829295+0000","last_became_peered":"2024-03-29T04:08:53.829295+0000","last_unstale":"2024-03-29T04:08:53.831543+0000","last_undegraded":"2024-03-29T04:08:53.831543+0000","last_fullsized":"2024-03-29T04:08:53.831543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:39.590764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823281+0000","last_change":"2024-03-29T04:08:53.819700+0000","last_active":"2024-03-29T04:08:53.823281+0000","last_peered":"2024-03-29T04:08:53.823281+0000","last_clean":"2024-03-29T04:08:53.823281+0000","last_became_active":"2024-03-29T04:08:53.819174+0000","last_became_peered":"2024-03-29T04:08:53.819174+0000","last_unstale":"2024-03-29T04:08:53.823281+0000","last_undegraded":"2024-03-29T04:08:53.823281+0000","last_fullsized":"2024-03-29T04:08:53.823281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811825+0000","last_change":"2024-03-29T04:08:53.812928+0000","last_active":"2024-03-29T04:08:54.811825+0000","last_peered":"2024-03-29T04:08:54.811825+0000","last_clean":"2024-03-29T04:08:54.811825+0000","last_became_active":"2024-03-29T04:08:53.812536+0000","last_became_peered":"2024-03-29T04:08:53.812536+0000","last_unstale":"2024-03-29T04:08:54.811825+0000","last_undegraded":"2024-03-29T04:08:54.811825+0000","last_fullsized":"2024-03-29T04:08:54.811825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814066+0000","last_change":"2024-03-29T04:08:53.823468+0000","last_active":"2024-03-29T04:08:54.814066+0000","last_peered":"2024-03-29T04:08:54.814066+0000","last_clean":"2024-03-29T04:08:54.814066+0000","last_became_active":"2024-03-29T04:08:53.822972+0000","last_became_peered":"2024-03-29T04:08:53.822972+0000","last_unstale":"2024-03-29T04:08:54.814066+0000","last_undegraded":"2024-03-29T04:08:54.814066+0000","last_fullsized":"2024-03-29T04:08:54.814066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815690+0000","last_change":"2024-03-29T04:08:53.816411+0000","last_active":"2024-03-29T04:08:54.815690+0000","last_peered":"2024-03-29T04:08:54.815690+0000","last_clean":"2024-03-29T04:08:54.815690+0000","last_became_active":"2024-03-29T04:08:53.816126+0000","last_became_peered":"2024-03-29T04:08:53.816126+0000","last_unstale":"2024-03-29T04:08:54.815690+0000","last_undegraded":"2024-03-29T04:08:54.815690+0000","last_fullsized":"2024-03-29T04:08:54.815690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.825587+0000","last_change":"2024-03-29T04:08:53.824360+0000","last_active":"2024-03-29T04:08:53.825587+0000","last_peered":"2024-03-29T04:08:53.825587+0000","last_clean":"2024-03-29T04:08:53.825587+0000","last_became_active":"2024-03-29T04:08:53.823207+0000","last_became_peered":"2024-03-29T04:08:53.823207+0000","last_unstale":"2024-03-29T04:08:53.825587+0000","last_undegraded":"2024-03-29T04:08:53.825587+0000","last_fullsized":"2024-03-29T04:08:53.825587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546389+0000","last_change":"2024-03-29T04:08:53.822584+0000","last_active":"2024-03-29T04:08:55.546389+0000","last_peered":"2024-03-29T04:08:55.546389+0000","last_clean":"2024-03-29T04:08:55.546389+0000","last_became_active":"2024-03-29T04:08:53.821230+0000","last_became_peered":"2024-03-29T04:08:53.821230+0000","last_unstale":"2024-03-29T04:08:55.546389+0000","last_undegraded":"2024-03-29T04:08:55.546389+0000","last_fullsized":"2024-03-29T04:08:55.546389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816186+0000","last_change":"2024-03-29T04:08:53.828332+0000","last_active":"2024-03-29T04:08:54.816186+0000","last_peered":"2024-03-29T04:08:54.816186+0000","last_clean":"2024-03-29T04:08:54.816186+0000","last_became_active":"2024-03-29T04:08:53.827954+0000","last_became_peered":"2024-03-29T04:08:53.827954+0000","last_unstale":"2024-03-29T04:08:54.816186+0000","last_undegraded":"2024-03-29T04:08:54.816186+0000","last_fullsized":"2024-03-29T04:08:54.816186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823715+0000","last_change":"2024-03-29T04:08:53.821853+0000","last_active":"2024-03-29T04:08:53.823715+0000","last_peered":"2024-03-29T04:08:53.823715+0000","last_clean":"2024-03-29T04:08:53.823715+0000","last_became_active":"2024-03-29T04:08:53.820469+0000","last_became_peered":"2024-03-29T04:08:53.820469+0000","last_unstale":"2024-03-29T04:08:53.823715+0000","last_undegraded":"2024-03-29T04:08:53.823715+0000","last_fullsized":"2024-03-29T04:08:53.823715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814489+0000","last_change":"2024-03-29T04:08:53.818641+0000","last_active":"2024-03-29T04:08:54.814489+0000","last_peered":"2024-03-29T04:08:54.814489+0000","last_clean":"2024-03-29T04:08:54.814489+0000","last_became_active":"2024-03-29T04:08:53.818082+0000","last_became_peered":"2024-03-29T04:08:53.818082+0000","last_unstale":"2024-03-29T04:08:54.814489+0000","last_undegraded":"2024-03-29T04:08:54.814489+0000","last_fullsized":"2024-03-29T04:08:54.814489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823823+0000","last_change":"2024-03-29T04:08:53.816408+0000","last_active":"2024-03-29T04:08:53.823823+0000","last_peered":"2024-03-29T04:08:53.823823+0000","last_clean":"2024-03-29T04:08:53.823823+0000","last_became_active":"2024-03-29T04:08:53.816127+0000","last_became_peered":"2024-03-29T04:08:53.816127+0000","last_unstale":"2024-03-29T04:08:53.823823+0000","last_undegraded":"2024-03-29T04:08:53.823823+0000","last_fullsized":"2024-03-29T04:08:53.823823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:18.995986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814214+0000","last_change":"2024-03-29T04:08:53.820680+0000","last_active":"2024-03-29T04:08:54.814214+0000","last_peered":"2024-03-29T04:08:54.814214+0000","last_clean":"2024-03-29T04:08:54.814214+0000","last_became_active":"2024-03-29T04:08:53.819878+0000","last_became_peered":"2024-03-29T04:08:53.819878+0000","last_unstale":"2024-03-29T04:08:54.814214+0000","last_undegraded":"2024-03-29T04:08:54.814214+0000","last_fullsized":"2024-03-29T04:08:54.814214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.819987+0000","last_change":"2024-03-29T04:08:53.816044+0000","last_active":"2024-03-29T04:08:53.819987+0000","last_peered":"2024-03-29T04:08:53.819987+0000","last_clean":"2024-03-29T04:08:53.819987+0000","last_became_active":"2024-03-29T04:08:53.815744+0000","last_became_peered":"2024-03-29T04:08:53.815744+0000","last_unstale":"2024-03-29T04:08:53.819987+0000","last_undegraded":"2024-03-29T04:08:53.819987+0000","last_fullsized":"2024-03-29T04:08:53.819987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:16.901767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816521+0000","last_change":"2024-03-29T04:08:53.830594+0000","last_active":"2024-03-29T04:08:54.816521+0000","last_peered":"2024-03-29T04:08:54.816521+0000","last_clean":"2024-03-29T04:08:54.816521+0000","last_became_active":"2024-03-29T04:08:53.830266+0000","last_became_peered":"2024-03-29T04:08:53.830266+0000","last_unstale":"2024-03-29T04:08:54.816521+0000","last_undegraded":"2024-03-29T04:08:54.816521+0000","last_fullsized":"2024-03-29T04:08:54.816521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812076+0000","last_change":"2024-03-29T04:08:53.817458+0000","last_active":"2024-03-29T04:08:54.812076+0000","last_peered":"2024-03-29T04:08:54.812076+0000","last_clean":"2024-03-29T04:08:54.812076+0000","last_became_active":"2024-03-29T04:08:53.817179+0000","last_became_peered":"2024-03-29T04:08:53.817179+0000","last_unstale":"2024-03-29T04:08:54.812076+0000","last_undegraded":"2024-03-29T04:08:54.812076+0000","last_fullsized":"2024-03-29T04:08:54.812076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823651+0000","last_change":"2024-03-29T04:08:53.819794+0000","last_active":"2024-03-29T04:08:53.823651+0000","last_peered":"2024-03-29T04:08:53.823651+0000","last_clean":"2024-03-29T04:08:53.823651+0000","last_became_active":"2024-03-29T04:08:53.819541+0000","last_became_peered":"2024-03-29T04:08:53.819541+0000","last_unstale":"2024-03-29T04:08:53.823651+0000","last_undegraded":"2024-03-29T04:08:53.823651+0000","last_fullsized":"2024-03-29T04:08:53.823651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823889+0000","last_change":"2024-03-29T04:08:53.823205+0000","last_active":"2024-03-29T04:08:53.823889+0000","last_peered":"2024-03-29T04:08:53.823889+0000","last_clean":"2024-03-29T04:08:53.823889+0000","last_became_active":"2024-03-29T04:08:53.822900+0000","last_became_peered":"2024-03-29T04:08:53.822900+0000","last_unstale":"2024-03-29T04:08:53.823889+0000","last_undegraded":"2024-03-29T04:08:53.823889+0000","last_fullsized":"2024-03-29T04:08:53.823889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:39.294386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814624+0000","last_change":"2024-03-29T04:08:53.821709+0000","last_active":"2024-03-29T04:08:54.814624+0000","last_peered":"2024-03-29T04:08:54.814624+0000","last_clean":"2024-03-29T04:08:54.814624+0000","last_became_active":"2024-03-29T04:08:53.820828+0000","last_became_peered":"2024-03-29T04:08:53.820828+0000","last_unstale":"2024-03-29T04:08:54.814624+0000","last_undegraded":"2024-03-29T04:08:54.814624+0000","last_fullsized":"2024-03-29T04:08:54.814624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.821137+0000","last_change":"2024-03-29T04:08:53.820303+0000","last_active":"2024-03-29T04:08:53.821137+0000","last_peered":"2024-03-29T04:08:53.821137+0000","last_clean":"2024-03-29T04:08:53.821137+0000","last_became_active":"2024-03-29T04:08:53.820024+0000","last_became_peered":"2024-03-29T04:08:53.820024+0000","last_unstale":"2024-03-29T04:08:53.821137+0000","last_undegraded":"2024-03-29T04:08:53.821137+0000","last_fullsized":"2024-03-29T04:08:53.821137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815813+0000","last_change":"2024-03-29T04:08:53.824803+0000","last_active":"2024-03-29T04:08:54.815813+0000","last_peered":"2024-03-29T04:08:54.815813+0000","last_clean":"2024-03-29T04:08:54.815813+0000","last_became_active":"2024-03-29T04:08:53.824454+0000","last_became_peered":"2024-03-29T04:08:53.824454+0000","last_unstale":"2024-03-29T04:08:54.815813+0000","last_undegraded":"2024-03-29T04:08:54.815813+0000","last_fullsized":"2024-03-29T04:08:54.815813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813758+0000","last_change":"2024-03-29T04:08:53.819297+0000","last_active":"2024-03-29T04:08:54.813758+0000","last_peered":"2024-03-29T04:08:54.813758+0000","last_clean":"2024-03-29T04:08:54.813758+0000","last_became_active":"2024-03-29T04:08:53.819016+0000","last_became_peered":"2024-03-29T04:08:53.819016+0000","last_unstale":"2024-03-29T04:08:54.813758+0000","last_undegraded":"2024-03-29T04:08:54.813758+0000","last_fullsized":"2024-03-29T04:08:54.813758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.830662+0000","last_change":"2024-03-29T04:08:52.815351+0000","last_active":"2024-03-29T04:08:53.830662+0000","last_peered":"2024-03-29T04:08:53.830662+0000","last_clean":"2024-03-29T04:08:53.830662+0000","last_became_active":"2024-03-29T04:08:52.814942+0000","last_became_peered":"2024-03-29T04:08:52.814942+0000","last_unstale":"2024-03-29T04:08:53.830662+0000","last_undegraded":"2024-03-29T04:08:53.830662+0000","last_fullsized":"2024-03-29T04:08:53.830662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823606+0000","last_change":"2024-03-29T04:08:53.820823+0000","last_active":"2024-03-29T04:08:53.823606+0000","last_peered":"2024-03-29T04:08:53.823606+0000","last_clean":"2024-03-29T04:08:53.823606+0000","last_became_active":"2024-03-29T04:08:53.820429+0000","last_became_peered":"2024-03-29T04:08:53.820429+0000","last_unstale":"2024-03-29T04:08:53.823606+0000","last_undegraded":"2024-03-29T04:08:53.823606+0000","last_fullsized":"2024-03-29T04:08:53.823606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815991+0000","last_change":"2024-03-29T04:08:52.812694+0000","last_active":"2024-03-29T04:08:54.815991+0000","last_peered":"2024-03-29T04:08:54.815991+0000","last_clean":"2024-03-29T04:08:54.815991+0000","last_became_active":"2024-03-29T04:08:52.812429+0000","last_became_peered":"2024-03-29T04:08:52.812429+0000","last_unstale":"2024-03-29T04:08:54.815991+0000","last_undegraded":"2024-03-29T04:08:54.815991+0000","last_fullsized":"2024-03-29T04:08:54.815991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.808199+0000","last_change":"2024-03-29T04:08:52.809190+0000","last_active":"2024-03-29T04:08:53.808199+0000","last_peered":"2024-03-29T04:08:53.808199+0000","last_clean":"2024-03-29T04:08:53.808199+0000","last_became_active":"2024-03-29T04:08:52.808923+0000","last_became_peered":"2024-03-29T04:08:52.808923+0000","last_unstale":"2024-03-29T04:08:53.808199+0000","last_undegraded":"2024-03-29T04:08:53.808199+0000","last_fullsized":"2024-03-29T04:08:53.808199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831108+0000","last_change":"2024-03-29T04:08:53.828865+0000","last_active":"2024-03-29T04:08:53.831108+0000","last_peered":"2024-03-29T04:08:53.831108+0000","last_clean":"2024-03-29T04:08:53.831108+0000","last_became_active":"2024-03-29T04:08:53.828601+0000","last_became_peered":"2024-03-29T04:08:53.828601+0000","last_unstale":"2024-03-29T04:08:53.831108+0000","last_undegraded":"2024-03-29T04:08:53.831108+0000","last_fullsized":"2024-03-29T04:08:53.831108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:23.273277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814227+0000","last_change":"2024-03-29T04:08:52.810336+0000","last_active":"2024-03-29T04:08:54.814227+0000","last_peered":"2024-03-29T04:08:54.814227+0000","last_clean":"2024-03-29T04:08:54.814227+0000","last_became_active":"2024-03-29T04:08:52.810021+0000","last_became_peered":"2024-03-29T04:08:52.810021+0000","last_unstale":"2024-03-29T04:08:54.814227+0000","last_undegraded":"2024-03-29T04:08:54.814227+0000","last_fullsized":"2024-03-29T04:08:54.814227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811960+0000","last_change":"2024-03-29T04:08:53.817702+0000","last_active":"2024-03-29T04:08:54.811960+0000","last_peered":"2024-03-29T04:08:54.811960+0000","last_clean":"2024-03-29T04:08:54.811960+0000","last_became_active":"2024-03-29T04:08:53.817117+0000","last_became_peered":"2024-03-29T04:08:53.817117+0000","last_unstale":"2024-03-29T04:08:54.811960+0000","last_undegraded":"2024-03-29T04:08:54.811960+0000","last_fullsized":"2024-03-29T04:08:54.811960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816011+0000","last_change":"2024-03-29T04:08:52.812898+0000","last_active":"2024-03-29T04:08:54.816011+0000","last_peered":"2024-03-29T04:08:54.816011+0000","last_clean":"2024-03-29T04:08:54.816011+0000","last_became_active":"2024-03-29T04:08:52.811143+0000","last_became_peered":"2024-03-29T04:08:52.811143+0000","last_unstale":"2024-03-29T04:08:54.816011+0000","last_undegraded":"2024-03-29T04:08:54.816011+0000","last_fullsized":"2024-03-29T04:08:54.816011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813759+0000","last_change":"2024-03-29T04:08:53.815139+0000","last_active":"2024-03-29T04:08:54.813759+0000","last_peered":"2024-03-29T04:08:54.813759+0000","last_clean":"2024-03-29T04:08:54.813759+0000","last_became_active":"2024-03-29T04:08:53.814836+0000","last_became_peered":"2024-03-29T04:08:53.814836+0000","last_unstale":"2024-03-29T04:08:54.813759+0000","last_undegraded":"2024-03-29T04:08:54.813759+0000","last_fullsized":"2024-03-29T04:08:54.813759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813822+0000","last_change":"2024-03-29T04:08:52.812383+0000","last_active":"2024-03-29T04:08:54.813822+0000","last_peered":"2024-03-29T04:08:54.813822+0000","last_clean":"2024-03-29T04:08:54.813822+0000","last_became_active":"2024-03-29T04:08:52.810343+0000","last_became_peered":"2024-03-29T04:08:52.810343+0000","last_unstale":"2024-03-29T04:08:54.813822+0000","last_undegraded":"2024-03-29T04:08:54.813822+0000","last_fullsized":"2024-03-29T04:08:54.813822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823577+0000","last_change":"2024-03-29T04:08:53.821898+0000","last_active":"2024-03-29T04:08:53.823577+0000","last_peered":"2024-03-29T04:08:53.823577+0000","last_clean":"2024-03-29T04:08:53.823577+0000","last_became_active":"2024-03-29T04:08:53.821126+0000","last_became_peered":"2024-03-29T04:08:53.821126+0000","last_unstale":"2024-03-29T04:08:53.823577+0000","last_undegraded":"2024-03-29T04:08:53.823577+0000","last_fullsized":"2024-03-29T04:08:53.823577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812079+0000","last_change":"2024-03-29T04:08:52.810835+0000","last_active":"2024-03-29T04:08:54.812079+0000","last_peered":"2024-03-29T04:08:54.812079+0000","last_clean":"2024-03-29T04:08:54.812079+0000","last_became_active":"2024-03-29T04:08:52.810585+0000","last_became_peered":"2024-03-29T04:08:52.810585+0000","last_unstale":"2024-03-29T04:08:54.812079+0000","last_undegraded":"2024-03-29T04:08:54.812079+0000","last_fullsized":"2024-03-29T04:08:54.812079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814427+0000","last_change":"2024-03-29T04:08:53.817718+0000","last_active":"2024-03-29T04:08:54.814427+0000","last_peered":"2024-03-29T04:08:54.814427+0000","last_clean":"2024-03-29T04:08:54.814427+0000","last_became_active":"2024-03-29T04:08:53.817419+0000","last_became_peered":"2024-03-29T04:08:53.817419+0000","last_unstale":"2024-03-29T04:08:54.814427+0000","last_undegraded":"2024-03-29T04:08:54.814427+0000","last_fullsized":"2024-03-29T04:08:54.814427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816294+0000","last_change":"2024-03-29T04:08:52.809428+0000","last_active":"2024-03-29T04:08:54.816294+0000","last_peered":"2024-03-29T04:08:54.816294+0000","last_clean":"2024-03-29T04:08:54.816294+0000","last_became_active":"2024-03-29T04:08:52.809124+0000","last_became_peered":"2024-03-29T04:08:52.809124+0000","last_unstale":"2024-03-29T04:08:54.816294+0000","last_undegraded":"2024-03-29T04:08:54.816294+0000","last_fullsized":"2024-03-29T04:08:54.816294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546068+0000","last_change":"2024-03-29T04:08:53.823456+0000","last_active":"2024-03-29T04:08:55.546068+0000","last_peered":"2024-03-29T04:08:55.546068+0000","last_clean":"2024-03-29T04:08:55.546068+0000","last_became_active":"2024-03-29T04:08:53.823196+0000","last_became_peered":"2024-03-29T04:08:53.823196+0000","last_unstale":"2024-03-29T04:08:55.546068+0000","last_undegraded":"2024-03-29T04:08:55.546068+0000","last_fullsized":"2024-03-29T04:08:55.546068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.828278+0000","last_change":"2024-03-29T04:08:52.804947+0000","last_active":"2024-03-29T04:08:53.828278+0000","last_peered":"2024-03-29T04:08:53.828278+0000","last_clean":"2024-03-29T04:08:53.828278+0000","last_became_active":"2024-03-29T04:08:52.804675+0000","last_became_peered":"2024-03-29T04:08:52.804675+0000","last_unstale":"2024-03-29T04:08:53.828278+0000","last_undegraded":"2024-03-29T04:08:53.828278+0000","last_fullsized":"2024-03-29T04:08:53.828278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823262+0000","last_change":"2024-03-29T04:08:53.815297+0000","last_active":"2024-03-29T04:08:53.823262+0000","last_peered":"2024-03-29T04:08:53.823262+0000","last_clean":"2024-03-29T04:08:53.823262+0000","last_became_active":"2024-03-29T04:08:53.815003+0000","last_became_peered":"2024-03-29T04:08:53.815003+0000","last_unstale":"2024-03-29T04:08:53.823262+0000","last_undegraded":"2024-03-29T04:08:53.823262+0000","last_fullsized":"2024-03-29T04:08:53.823262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:16.901767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814596+0000","last_change":"2024-03-29T04:08:52.809942+0000","last_active":"2024-03-29T04:08:54.814596+0000","last_peered":"2024-03-29T04:08:54.814596+0000","last_clean":"2024-03-29T04:08:54.814596+0000","last_became_active":"2024-03-29T04:08:52.809671+0000","last_became_peered":"2024-03-29T04:08:52.809671+0000","last_unstale":"2024-03-29T04:08:54.814596+0000","last_undegraded":"2024-03-29T04:08:54.814596+0000","last_fullsized":"2024-03-29T04:08:54.814596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546150+0000","last_change":"2024-03-29T04:08:53.818585+0000","last_active":"2024-03-29T04:08:55.546150+0000","last_peered":"2024-03-29T04:08:55.546150+0000","last_clean":"2024-03-29T04:08:55.546150+0000","last_became_active":"2024-03-29T04:08:53.818252+0000","last_became_peered":"2024-03-29T04:08:53.818252+0000","last_unstale":"2024-03-29T04:08:55.546150+0000","last_undegraded":"2024-03-29T04:08:55.546150+0000","last_fullsized":"2024-03-29T04:08:55.546150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815950+0000","last_change":"2024-03-29T04:08:52.810780+0000","last_active":"2024-03-29T04:08:54.815950+0000","last_peered":"2024-03-29T04:08:54.815950+0000","last_clean":"2024-03-29T04:08:54.815950+0000","last_became_active":"2024-03-29T04:08:52.810487+0000","last_became_peered":"2024-03-29T04:08:52.810487+0000","last_unstale":"2024-03-29T04:08:54.815950+0000","last_undegraded":"2024-03-29T04:08:54.815950+0000","last_fullsized":"2024-03-29T04:08:54.815950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816134+0000","last_change":"2024-03-29T04:08:53.825665+0000","last_active":"2024-03-29T04:08:54.816134+0000","last_peered":"2024-03-29T04:08:54.816134+0000","last_clean":"2024-03-29T04:08:54.816134+0000","last_became_active":"2024-03-29T04:08:53.825229+0000","last_became_peered":"2024-03-29T04:08:53.825229+0000","last_unstale":"2024-03-29T04:08:54.816134+0000","last_undegraded":"2024-03-29T04:08:54.816134+0000","last_fullsized":"2024-03-29T04:08:54.816134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.830026+0000","last_change":"2024-03-29T04:08:52.807491+0000","last_active":"2024-03-29T04:08:53.830026+0000","last_peered":"2024-03-29T04:08:53.830026+0000","last_clean":"2024-03-29T04:08:53.830026+0000","last_became_active":"2024-03-29T04:08:52.807236+0000","last_became_peered":"2024-03-29T04:08:52.807236+0000","last_unstale":"2024-03-29T04:08:53.830026+0000","last_undegraded":"2024-03-29T04:08:53.830026+0000","last_fullsized":"2024-03-29T04:08:53.830026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545849+0000","last_change":"2024-03-29T04:08:53.819664+0000","last_active":"2024-03-29T04:08:55.545849+0000","last_peered":"2024-03-29T04:08:55.545849+0000","last_clean":"2024-03-29T04:08:55.545849+0000","last_became_active":"2024-03-29T04:08:53.819414+0000","last_became_peered":"2024-03-29T04:08:53.819414+0000","last_unstale":"2024-03-29T04:08:55.545849+0000","last_undegraded":"2024-03-29T04:08:55.545849+0000","last_fullsized":"2024-03-29T04:08:55.545849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815063+0000","last_change":"2024-03-29T04:08:52.806445+0000","last_active":"2024-03-29T04:08:54.815063+0000","last_peered":"2024-03-29T04:08:54.815063+0000","last_clean":"2024-03-29T04:08:54.815063+0000","last_became_active":"2024-03-29T04:08:52.806166+0000","last_became_peered":"2024-03-29T04:08:52.806166+0000","last_unstale":"2024-03-29T04:08:54.815063+0000","last_undegraded":"2024-03-29T04:08:54.815063+0000","last_fullsized":"2024-03-29T04:08:54.815063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:03.546395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546168+0000","last_change":"2024-03-29T04:08:53.819783+0000","last_active":"2024-03-29T04:08:55.546168+0000","last_peered":"2024-03-29T04:08:55.546168+0000","last_clean":"2024-03-29T04:08:55.546168+0000","last_became_active":"2024-03-29T04:08:53.819507+0000","last_became_peered":"2024-03-29T04:08:53.819507+0000","last_unstale":"2024-03-29T04:08:55.546168+0000","last_undegraded":"2024-03-29T04:08:55.546168+0000","last_fullsized":"2024-03-29T04:08:55.546168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831487+0000","last_change":"2024-03-29T04:08:52.812349+0000","last_active":"2024-03-29T04:08:53.831487+0000","last_peered":"2024-03-29T04:08:53.831487+0000","last_clean":"2024-03-29T04:08:53.831487+0000","last_became_active":"2024-03-29T04:08:52.812081+0000","last_became_peered":"2024-03-29T04:08:52.812081+0000","last_unstale":"2024-03-29T04:08:53.831487+0000","last_undegraded":"2024-03-29T04:08:53.831487+0000","last_fullsized":"2024-03-29T04:08:53.831487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546387+0000","last_change":"2024-03-29T04:08:53.827351+0000","last_active":"2024-03-29T04:08:55.546387+0000","last_peered":"2024-03-29T04:08:55.546387+0000","last_clean":"2024-03-29T04:08:55.546387+0000","last_became_active":"2024-03-29T04:08:53.826255+0000","last_became_peered":"2024-03-29T04:08:53.826255+0000","last_unstale":"2024-03-29T04:08:55.546387+0000","last_undegraded":"2024-03-29T04:08:55.546387+0000","last_fullsized":"2024-03-29T04:08:55.546387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812164+0000","last_change":"2024-03-29T04:08:52.805752+0000","last_active":"2024-03-29T04:08:54.812164+0000","last_peered":"2024-03-29T04:08:54.812164+0000","last_clean":"2024-03-29T04:08:54.812164+0000","last_became_active":"2024-03-29T04:08:52.804921+0000","last_became_peered":"2024-03-29T04:08:52.804921+0000","last_unstale":"2024-03-29T04:08:54.812164+0000","last_undegraded":"2024-03-29T04:08:54.812164+0000","last_fullsized":"2024-03-29T04:08:54.812164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815243+0000","last_change":"2024-03-29T04:08:53.821399+0000","last_active":"2024-03-29T04:08:54.815243+0000","last_peered":"2024-03-29T04:08:54.815243+0000","last_clean":"2024-03-29T04:08:54.815243+0000","last_became_active":"2024-03-29T04:08:53.821026+0000","last_became_peered":"2024-03-29T04:08:53.821026+0000","last_unstale":"2024-03-29T04:08:54.815243+0000","last_undegraded":"2024-03-29T04:08:54.815243+0000","last_fullsized":"2024-03-29T04:08:54.815243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.809520+0000","last_change":"2024-03-29T04:08:52.811397+0000","last_active":"2024-03-29T04:08:53.809520+0000","last_peered":"2024-03-29T04:08:53.809520+0000","last_clean":"2024-03-29T04:08:53.809520+0000","last_became_active":"2024-03-29T04:08:52.810968+0000","last_became_peered":"2024-03-29T04:08:52.810968+0000","last_unstale":"2024-03-29T04:08:53.809520+0000","last_undegraded":"2024-03-29T04:08:53.809520+0000","last_fullsized":"2024-03-29T04:08:53.809520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546224+0000","last_change":"2024-03-29T04:08:53.824444+0000","last_active":"2024-03-29T04:08:55.546224+0000","last_peered":"2024-03-29T04:08:55.546224+0000","last_clean":"2024-03-29T04:08:55.546224+0000","last_became_active":"2024-03-29T04:08:53.821578+0000","last_became_peered":"2024-03-29T04:08:53.821578+0000","last_unstale":"2024-03-29T04:08:55.546224+0000","last_undegraded":"2024-03-29T04:08:55.546224+0000","last_fullsized":"2024-03-29T04:08:55.546224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545812+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:08:55.545812+0000","last_peered":"2024-03-29T04:08:55.545812+0000","last_clean":"2024-03-29T04:08:55.545812+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:08:55.545812+0000","last_undegraded":"2024-03-29T04:08:55.545812+0000","last_fullsized":"2024-03-29T04:08:55.545812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813649+0000","last_change":"2024-03-29T04:08:52.807738+0000","last_active":"2024-03-29T04:08:54.813649+0000","last_peered":"2024-03-29T04:08:54.813649+0000","last_clean":"2024-03-29T04:08:54.813649+0000","last_became_active":"2024-03-29T04:08:52.807442+0000","last_became_peered":"2024-03-29T04:08:52.807442+0000","last_unstale":"2024-03-29T04:08:54.813649+0000","last_undegraded":"2024-03-29T04:08:54.813649+0000","last_fullsized":"2024-03-29T04:08:54.813649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.826424+0000","last_change":"2024-03-29T04:08:53.825630+0000","last_active":"2024-03-29T04:08:53.826424+0000","last_peered":"2024-03-29T04:08:53.826424+0000","last_clean":"2024-03-29T04:08:53.826424+0000","last_became_active":"2024-03-29T04:08:53.825386+0000","last_became_peered":"2024-03-29T04:08:53.825386+0000","last_unstale":"2024-03-29T04:08:53.826424+0000","last_undegraded":"2024-03-29T04:08:53.826424+0000","last_fullsized":"2024-03-29T04:08:53.826424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545708+0000","last_change":"2024-03-29T04:08:52.820187+0000","last_active":"2024-03-29T04:08:55.545708+0000","last_peered":"2024-03-29T04:08:55.545708+0000","last_clean":"2024-03-29T04:08:55.545708+0000","last_became_active":"2024-03-29T04:08:52.819912+0000","last_became_peered":"2024-03-29T04:08:52.819912+0000","last_unstale":"2024-03-29T04:08:55.545708+0000","last_undegraded":"2024-03-29T04:08:55.545708+0000","last_fullsized":"2024-03-29T04:08:55.545708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823307+0000","last_change":"2024-03-29T04:08:53.815607+0000","last_active":"2024-03-29T04:08:53.823307+0000","last_peered":"2024-03-29T04:08:53.823307+0000","last_clean":"2024-03-29T04:08:53.823307+0000","last_became_active":"2024-03-29T04:08:53.814274+0000","last_became_peered":"2024-03-29T04:08:53.814274+0000","last_unstale":"2024-03-29T04:08:53.823307+0000","last_undegraded":"2024-03-29T04:08:53.823307+0000","last_fullsized":"2024-03-29T04:08:53.823307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:18.995986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.809640+0000","last_change":"2024-03-29T04:08:52.813143+0000","last_active":"2024-03-29T04:08:53.809640+0000","last_peered":"2024-03-29T04:08:53.809640+0000","last_clean":"2024-03-29T04:08:53.809640+0000","last_became_active":"2024-03-29T04:08:52.812866+0000","last_became_peered":"2024-03-29T04:08:52.812866+0000","last_unstale":"2024-03-29T04:08:53.809640+0000","last_undegraded":"2024-03-29T04:08:53.809640+0000","last_fullsized":"2024-03-29T04:08:53.809640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811826+0000","last_change":"2024-03-29T04:08:53.818488+0000","last_active":"2024-03-29T04:08:54.811826+0000","last_peered":"2024-03-29T04:08:54.811826+0000","last_clean":"2024-03-29T04:08:54.811826+0000","last_became_active":"2024-03-29T04:08:53.818222+0000","last_became_peered":"2024-03-29T04:08:53.818222+0000","last_unstale":"2024-03-29T04:08:54.811826+0000","last_undegraded":"2024-03-29T04:08:54.811826+0000","last_fullsized":"2024-03-29T04:08:54.811826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814714+0000","last_change":"2024-03-29T04:08:52.808830+0000","last_active":"2024-03-29T04:08:54.814714+0000","last_peered":"2024-03-29T04:08:54.814714+0000","last_clean":"2024-03-29T04:08:54.814714+0000","last_became_active":"2024-03-29T04:08:52.808524+0000","last_became_peered":"2024-03-29T04:08:52.808524+0000","last_unstale":"2024-03-29T04:08:54.814714+0000","last_undegraded":"2024-03-29T04:08:54.814714+0000","last_fullsized":"2024-03-29T04:08:54.814714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815259+0000","last_change":"2024-03-29T04:08:53.823062+0000","last_active":"2024-03-29T04:08:54.815259+0000","last_peered":"2024-03-29T04:08:54.815259+0000","last_clean":"2024-03-29T04:08:54.815259+0000","last_became_active":"2024-03-29T04:08:53.822666+0000","last_became_peered":"2024-03-29T04:08:53.822666+0000","last_unstale":"2024-03-29T04:08:54.815259+0000","last_undegraded":"2024-03-29T04:08:54.815259+0000","last_fullsized":"2024-03-29T04:08:54.815259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812019+0000","last_change":"2024-03-29T04:08:52.799226+0000","last_active":"2024-03-29T04:08:54.812019+0000","last_peered":"2024-03-29T04:08:54.812019+0000","last_clean":"2024-03-29T04:08:54.812019+0000","last_became_active":"2024-03-29T04:08:52.798964+0000","last_became_peered":"2024-03-29T04:08:52.798964+0000","last_unstale":"2024-03-29T04:08:54.812019+0000","last_undegraded":"2024-03-29T04:08:54.812019+0000","last_fullsized":"2024-03-29T04:08:54.812019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815451+0000","last_change":"2024-03-29T04:08:53.821737+0000","last_active":"2024-03-29T04:08:54.815451+0000","last_peered":"2024-03-29T04:08:54.815451+0000","last_clean":"2024-03-29T04:08:54.815451+0000","last_became_active":"2024-03-29T04:08:53.820680+0000","last_became_peered":"2024-03-29T04:08:53.820680+0000","last_unstale":"2024-03-29T04:08:54.815451+0000","last_undegraded":"2024-03-29T04:08:54.815451+0000","last_fullsized":"2024-03-29T04:08:54.815451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546333+0000","last_change":"2024-03-29T04:08:52.821118+0000","last_active":"2024-03-29T04:08:55.546333+0000","last_peered":"2024-03-29T04:08:55.546333+0000","last_clean":"2024-03-29T04:08:55.546333+0000","last_became_active":"2024-03-29T04:08:52.820549+0000","last_became_peered":"2024-03-29T04:08:52.820549+0000","last_unstale":"2024-03-29T04:08:55.546333+0000","last_undegraded":"2024-03-29T04:08:55.546333+0000","last_fullsized":"2024-03-29T04:08:55.546333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811978+0000","last_change":"2024-03-29T04:08:53.810781+0000","last_active":"2024-03-29T04:08:54.811978+0000","last_peered":"2024-03-29T04:08:54.811978+0000","last_clean":"2024-03-29T04:08:54.811978+0000","last_became_active":"2024-03-29T04:08:53.810518+0000","last_became_peered":"2024-03-29T04:08:53.810518+0000","last_unstale":"2024-03-29T04:08:54.811978+0000","last_undegraded":"2024-03-29T04:08:54.811978+0000","last_fullsized":"2024-03-29T04:08:54.811978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816447+0000","last_change":"2024-03-29T04:08:52.809876+0000","last_active":"2024-03-29T04:08:54.816447+0000","last_peered":"2024-03-29T04:08:54.816447+0000","last_clean":"2024-03-29T04:08:54.816447+0000","last_became_active":"2024-03-29T04:08:52.809575+0000","last_became_peered":"2024-03-29T04:08:52.809575+0000","last_unstale":"2024-03-29T04:08:54.816447+0000","last_undegraded":"2024-03-29T04:08:54.816447+0000","last_fullsized":"2024-03-29T04:08:54.816447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.832275+0000","last_change":"2024-03-29T04:08:53.831225+0000","last_active":"2024-03-29T04:08:53.832275+0000","last_peered":"2024-03-29T04:08:53.832275+0000","last_clean":"2024-03-29T04:08:53.832275+0000","last_became_active":"2024-03-29T04:08:53.830819+0000","last_became_peered":"2024-03-29T04:08:53.830819+0000","last_unstale":"2024-03-29T04:08:53.832275+0000","last_undegraded":"2024-03-29T04:08:53.832275+0000","last_fullsized":"2024-03-29T04:08:53.832275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816132+0000","last_change":"2024-03-29T04:08:52.812725+0000","last_active":"2024-03-29T04:08:54.816132+0000","last_peered":"2024-03-29T04:08:54.816132+0000","last_clean":"2024-03-29T04:08:54.816132+0000","last_became_active":"2024-03-29T04:08:52.812020+0000","last_became_peered":"2024-03-29T04:08:52.812020+0000","last_unstale":"2024-03-29T04:08:54.816132+0000","last_undegraded":"2024-03-29T04:08:54.816132+0000","last_fullsized":"2024-03-29T04:08:54.816132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.816359+0000","last_change":"2024-03-29T04:08:53.811809+0000","last_active":"2024-03-29T04:08:53.816359+0000","last_peered":"2024-03-29T04:08:53.816359+0000","last_clean":"2024-03-29T04:08:53.816359+0000","last_became_active":"2024-03-29T04:08:53.811513+0000","last_became_peered":"2024-03-29T04:08:53.811513+0000","last_unstale":"2024-03-29T04:08:53.816359+0000","last_undegraded":"2024-03-29T04:08:53.816359+0000","last_fullsized":"2024-03-29T04:08:53.816359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:59.816889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545699+0000","last_change":"2024-03-29T04:08:52.814972+0000","last_active":"2024-03-29T04:08:55.545699+0000","last_peered":"2024-03-29T04:08:55.545699+0000","last_clean":"2024-03-29T04:08:55.545699+0000","last_became_active":"2024-03-29T04:08:52.814714+0000","last_became_peered":"2024-03-29T04:08:52.814714+0000","last_unstale":"2024-03-29T04:08:55.545699+0000","last_undegraded":"2024-03-29T04:08:55.545699+0000","last_fullsized":"2024-03-29T04:08:55.545699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814365+0000","last_change":"2024-03-29T04:08:53.817677+0000","last_active":"2024-03-29T04:08:54.814365+0000","last_peered":"2024-03-29T04:08:54.814365+0000","last_clean":"2024-03-29T04:08:54.814365+0000","last_became_active":"2024-03-29T04:08:53.815481+0000","last_became_peered":"2024-03-29T04:08:53.815481+0000","last_unstale":"2024-03-29T04:08:54.814365+0000","last_undegraded":"2024-03-29T04:08:54.814365+0000","last_fullsized":"2024-03-29T04:08:54.814365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811927+0000","last_change":"2024-03-29T04:08:52.814031+0000","last_active":"2024-03-29T04:08:54.811927+0000","last_peered":"2024-03-29T04:08:54.811927+0000","last_clean":"2024-03-29T04:08:54.811927+0000","last_became_active":"2024-03-29T04:08:52.813714+0000","last_became_peered":"2024-03-29T04:08:52.813714+0000","last_unstale":"2024-03-29T04:08:54.811927+0000","last_undegraded":"2024-03-29T04:08:54.811927+0000","last_fullsized":"2024-03-29T04:08:54.811927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823768+0000","last_change":"2024-03-29T04:08:53.818653+0000","last_active":"2024-03-29T04:08:53.823768+0000","last_peered":"2024-03-29T04:08:53.823768+0000","last_clean":"2024-03-29T04:08:53.823768+0000","last_became_active":"2024-03-29T04:08:53.818387+0000","last_became_peered":"2024-03-29T04:08:53.818387+0000","last_unstale":"2024-03-29T04:08:53.823768+0000","last_undegraded":"2024-03-29T04:08:53.823768+0000","last_fullsized":"2024-03-29T04:08:53.823768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812062+0000","last_change":"2024-03-29T04:08:52.806456+0000","last_active":"2024-03-29T04:08:54.812062+0000","last_peered":"2024-03-29T04:08:54.812062+0000","last_clean":"2024-03-29T04:08:54.812062+0000","last_became_active":"2024-03-29T04:08:52.806184+0000","last_became_peered":"2024-03-29T04:08:52.806184+0000","last_unstale":"2024-03-29T04:08:54.812062+0000","last_undegraded":"2024-03-29T04:08:54.812062+0000","last_fullsized":"2024-03-29T04:08:54.812062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831092+0000","last_change":"2024-03-29T04:08:53.828867+0000","last_active":"2024-03-29T04:08:53.831092+0000","last_peered":"2024-03-29T04:08:53.831092+0000","last_clean":"2024-03-29T04:08:53.831092+0000","last_became_active":"2024-03-29T04:08:53.828610+0000","last_became_peered":"2024-03-29T04:08:53.828610+0000","last_unstale":"2024-03-29T04:08:53.831092+0000","last_undegraded":"2024-03-29T04:08:53.831092+0000","last_fullsized":"2024-03-29T04:08:53.831092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546226+0000","last_change":"2024-03-29T04:08:52.821021+0000","last_active":"2024-03-29T04:08:55.546226+0000","last_peered":"2024-03-29T04:08:55.546226+0000","last_clean":"2024-03-29T04:08:55.546226+0000","last_became_active":"2024-03-29T04:08:52.820478+0000","last_became_peered":"2024-03-29T04:08:52.820478+0000","last_unstale":"2024-03-29T04:08:55.546226+0000","last_undegraded":"2024-03-29T04:08:55.546226+0000","last_fullsized":"2024-03-29T04:08:55.546226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815868+0000","last_change":"2024-03-29T04:08:53.825032+0000","last_active":"2024-03-29T04:08:54.815868+0000","last_peered":"2024-03-29T04:08:54.815868+0000","last_clean":"2024-03-29T04:08:54.815868+0000","last_became_active":"2024-03-29T04:08:53.824740+0000","last_became_peered":"2024-03-29T04:08:53.824740+0000","last_unstale":"2024-03-29T04:08:54.815868+0000","last_undegraded":"2024-03-29T04:08:54.815868+0000","last_fullsized":"2024-03-29T04:08:54.815868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815425+0000","last_change":"2024-03-29T04:08:53.816669+0000","last_active":"2024-03-29T04:08:54.815425+0000","last_peered":"2024-03-29T04:08:54.815425+0000","last_clean":"2024-03-29T04:08:54.815425+0000","last_became_active":"2024-03-29T04:08:53.816348+0000","last_became_peered":"2024-03-29T04:08:53.816348+0000","last_unstale":"2024-03-29T04:08:54.815425+0000","last_undegraded":"2024-03-29T04:08:54.815425+0000","last_fullsized":"2024-03-29T04:08:54.815425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545977+0000","last_change":"2024-03-29T04:08:52.820909+0000","last_active":"2024-03-29T04:08:55.545977+0000","last_peered":"2024-03-29T04:08:55.545977+0000","last_clean":"2024-03-29T04:08:55.545977+0000","last_became_active":"2024-03-29T04:08:52.820523+0000","last_became_peered":"2024-03-29T04:08:52.820523+0000","last_unstale":"2024-03-29T04:08:55.545977+0000","last_undegraded":"2024-03-29T04:08:55.545977+0000","last_fullsized":"2024-03-29T04:08:55.545977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815127+0000","last_change":"2024-03-29T04:08:53.817769+0000","last_active":"2024-03-29T04:08:54.815127+0000","last_peered":"2024-03-29T04:08:54.815127+0000","last_clean":"2024-03-29T04:08:54.815127+0000","last_became_active":"2024-03-29T04:08:53.816753+0000","last_became_peered":"2024-03-29T04:08:53.816753+0000","last_unstale":"2024-03-29T04:08:54.815127+0000","last_undegraded":"2024-03-29T04:08:54.815127+0000","last_fullsized":"2024-03-29T04:08:54.815127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546206+0000","last_change":"2024-03-29T04:08:52.820167+0000","last_active":"2024-03-29T04:08:55.546206+0000","last_peered":"2024-03-29T04:08:55.546206+0000","last_clean":"2024-03-29T04:08:55.546206+0000","last_became_active":"2024-03-29T04:08:52.819858+0000","last_became_peered":"2024-03-29T04:08:52.819858+0000","last_unstale":"2024-03-29T04:08:55.546206+0000","last_undegraded":"2024-03-29T04:08:55.546206+0000","last_fullsized":"2024-03-29T04:08:55.546206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.823596+0000","last_change":"2024-03-29T04:08:53.822001+0000","last_active":"2024-03-29T04:08:53.823596+0000","last_peered":"2024-03-29T04:08:53.823596+0000","last_clean":"2024-03-29T04:08:53.823596+0000","last_became_active":"2024-03-29T04:08:53.821669+0000","last_became_peered":"2024-03-29T04:08:53.821669+0000","last_unstale":"2024-03-29T04:08:53.823596+0000","last_undegraded":"2024-03-29T04:08:53.823596+0000","last_fullsized":"2024-03-29T04:08:53.823596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:23.273277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.830974+0000","last_change":"2024-03-29T04:08:52.807347+0000","last_active":"2024-03-29T04:08:53.830974+0000","last_peered":"2024-03-29T04:08:53.830974+0000","last_clean":"2024-03-29T04:08:53.830974+0000","last_became_active":"2024-03-29T04:08:52.807042+0000","last_became_peered":"2024-03-29T04:08:52.807042+0000","last_unstale":"2024-03-29T04:08:53.830974+0000","last_undegraded":"2024-03-29T04:08:53.830974+0000","last_fullsized":"2024-03-29T04:08:53.830974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.831375+0000","last_change":"2024-03-29T04:08:53.829560+0000","last_active":"2024-03-29T04:08:53.831375+0000","last_peered":"2024-03-29T04:08:53.831375+0000","last_clean":"2024-03-29T04:08:53.831375+0000","last_became_active":"2024-03-29T04:08:53.829312+0000","last_became_peered":"2024-03-29T04:08:53.829312+0000","last_unstale":"2024-03-29T04:08:53.831375+0000","last_undegraded":"2024-03-29T04:08:53.831375+0000","last_fullsized":"2024-03-29T04:08:53.831375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:39.294386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T04:08:53.829143+0000","last_change":"2024-03-29T04:08:52.813322+0000","last_active":"2024-03-29T04:08:53.829143+0000","last_peered":"2024-03-29T04:08:53.829143+0000","last_clean":"2024-03-29T04:08:53.829143+0000","last_became_active":"2024-03-29T04:08:52.813020+0000","last_became_peered":"2024-03-29T04:08:52.813020+0000","last_unstale":"2024-03-29T04:08:53.829143+0000","last_undegraded":"2024-03-29T04:08:53.829143+0000","last_fullsized":"2024-03-29T04:08:53.829143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":665435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":212992,"data_stored":69797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":68185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":208896,"data_stored":68345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":659191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:08:58.252 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T04:08:58.252 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-29T04:08:58.252 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-29T04:08:58.253 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-29T04:08:58.253 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-29T04:08:58.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.410+0000 7fd4fecb7640 1 Processor -- start 2024-03-29T04:08:58.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.411+0000 7fd4fecb7640 1 -- start start 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4fecb7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd4f80b1850 0x7fd4f80b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4fecb7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f80b32e0 0x7fd4f809aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4fecb7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f80b1d00 con 0x7fd4f815fba0 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4fecb7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd4f80b1fe0 con 0x7fd4f80b1850 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4fecb7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f80b22c0 con 0x7fd4f80b32e0 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd4f80b1850 0x7fd4f80b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd4f80b1850 0x7fd4f80b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45330/0 (socket says 172.21.15.156:45330) 2024-03-29T04:08:58.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1978635035 learned_addr learned my addr 172.21.15.156:0/1978635035 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:58.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4fd254640 1 --2- 172.21.15.156:0/1978635035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f80b32e0 0x7fd4f809aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1978635035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f80b32e0 msgr2=0x7fd4f809aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:58.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 --2- 172.21.15.156:0/1978635035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f80b32e0 0x7fd4f809aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1978635035 >> v1:172.21.15.203:6789/0 conn(0x7fd4f815fba0 legacy=0x7fd4f8160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.412+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1978635035 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f80b25a0 con 0x7fd4f80b1850 2024-03-29T04:08:58.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.413+0000 7fd4f7fff640 1 --2- 172.21.15.156:0/1978635035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b1850 0x7fd4f80b2da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009870 tx=0x7fd4ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283b885297f84bea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.413+0000 7fd4f77fe640 1 -- 172.21.15.156:0/1978635035 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec035020 con 0x7fd4f80b1850 2024-03-29T04:08:58.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.413+0000 7fd4f77fe640 1 -- 172.21.15.156:0/1978635035 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ec031640 con 0x7fd4f80b1850 2024-03-29T04:08:58.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.413+0000 7fd4f77fe640 1 -- 172.21.15.156:0/1978635035 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec02d040 con 0x7fd4f80b1850 2024-03-29T04:08:58.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1978635035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b1850 msgr2=0x7fd4f80b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1978635035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b1850 0x7fd4f80b2da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009870 tx=0x7fd4ec02b8f0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1978635035 shutdown_connections 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1978635035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f80b32e0 0x7fd4f809aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1978635035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b1850 0x7fd4f80b2da0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1978635035 >> 172.21.15.156:0/1978635035 conn(0x7fd4f80b3d90 msgr2=0x7fd4f80b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1978635035 shutdown_connections 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.414+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1978635035 wait complete. 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.415+0000 7fd4fecb7640 1 Processor -- start 2024-03-29T04:08:58.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.415+0000 7fd4fecb7640 1 -- start start 2024-03-29T04:08:58.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fecb7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4f80b1850 0x7fd4f81e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fecb7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 0x7fd4f815aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fecb7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f815fba0 0x7fd4f80c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fecb7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f809af80 con 0x7fd4f80b32e0 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fecb7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f80b3990 con 0x7fd4f80b1850 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fecb7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f809b100 con 0x7fd4f815fba0 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 0x7fd4f815aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 0x7fd4f815aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45332/0 (socket says 172.21.15.156:45332) 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1323591253 learned_addr learned my addr 172.21.15.156:0/1323591253 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fd254640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f815fba0 0x7fd4f80c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fca53640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4f80b1850 0x7fd4f81e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f815fba0 msgr2=0x7fd4f80c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f815fba0 0x7fd4f80c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4f80b1850 msgr2=0x7fd4f81e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4f80b1850 0x7fd4f81e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 -- 172.21.15.156:0/1323591253 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f806d560 con 0x7fd4f80b32e0 2024-03-29T04:08:58.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4fca53640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4f80b1850 0x7fd4f81e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f7fff640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 0x7fd4f815aee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec000c00 tx=0x7fd4ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.416+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec040070 con 0x7fd4f80b32e0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.417+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ec0399f0 con 0x7fd4f80b32e0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.417+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f806d840 con 0x7fd4f80b32e0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.417+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec035040 con 0x7fd4f80b32e0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.417+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f806e0e0 con 0x7fd4f80b32e0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.418+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f8068d20 con 0x7fd4f80b32e0 2024-03-29T04:08:58.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.418+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd4ec04b060 con 0x7fd4f80b32e0 2024-03-29T04:08:58.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.418+0000 7fd4f57fa640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd4e406c530 0x7fd4e406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.419+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd4ec0ace70 con 0x7fd4f80b32e0 2024-03-29T04:08:58.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.419+0000 7fd4fca53640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd4e406c530 0x7fd4e406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.419+0000 7fd4fca53640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd4e406c530 0x7fd4e406e9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd4f81f7860 tx=0x7fd4e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.423+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd4ec07b520 con 0x7fd4f80b32e0 2024-03-29T04:08:58.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.653+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd4f8073bf0 con 0x7fd4f80b32e0 2024-03-29T04:08:58.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.658+0000 7fd4f57fa640 1 -- 172.21.15.156:0/1323591253 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173739 (secure 0 0 0) 0x7fd4ec07ae20 con 0x7fd4f80b32e0 2024-03-29T04:08:58.640 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:58.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.660+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd4e406c530 msgr2=0x7fd4e406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.660+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd4e406c530 0x7fd4e406e9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd4f81f7860 tx=0x7fd4e8009040 comp rx=0 tx=0).stop 2024-03-29T04:08:58.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.660+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 msgr2=0x7fd4f815aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.660+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 0x7fd4f815aee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec000c00 tx=0x7fd4ec0046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 shutdown_connections 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd4e406c530 0x7fd4e406e9f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd4f815fba0 0x7fd4f80c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4f80b32e0 0x7fd4f815aee0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 --2- 172.21.15.156:0/1323591253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4f80b1850 0x7fd4f81e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 >> 172.21.15.156:0/1323591253 conn(0x7fd4f80b3d90 msgr2=0x7fd4f8070ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 shutdown_connections 2024-03-29T04:08:58.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.661+0000 7fd4fecb7640 1 -- 172.21.15.156:0/1323591253 wait complete. 2024-03-29T04:08:58.651 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2875364846},{"type":"v1","addr":"172.21.15.156:6833","nonce":2875364846}]},"active_addr":"172.21.15.156:6833/2875364846","active_change":"2024-03-29T04:08:33.626595+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":348519115}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":1911124671}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":4058372666}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":25446165}]}]} 2024-03-29T04:08:58.654 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-29T04:08:58.654 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-29T04:08:58.655 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:58.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.776+0000 7f10d0b11640 1 Processor -- start 2024-03-29T04:08:58.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.776+0000 7f10d0b11640 1 -- start start 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10d0b11640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f10cc15e720 0x7f10cc0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10d0b11640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f10cc0b40d0 0x7f10cc06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10d0b11640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10cc15eb00 con 0x7f10cc06baf0 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10d0b11640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f10cc0b5b30 con 0x7f10cc0b40d0 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10d0b11640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f10cc06bf40 con 0x7f10cc15e720 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f10cc0b40d0 0x7f10cc06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f10cc0b40d0 0x7f10cc06b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45334/0 (socket says 172.21.15.156:45334) 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 -- 172.21.15.156:0/3250406261 learned_addr learned my addr 172.21.15.156:0/3250406261 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10ca59c640 1 --2- 172.21.15.156:0/3250406261 >> v2:172.21.15.203:3300/0 conn(0x7f10cc15e720 0x7f10cc0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 -- 172.21.15.156:0/3250406261 >> v2:172.21.15.203:3300/0 conn(0x7f10cc15e720 msgr2=0x7f10cc0b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:58.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 --2- 172.21.15.156:0/3250406261 >> v2:172.21.15.203:3300/0 conn(0x7f10cc15e720 0x7f10cc0b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 -- 172.21.15.156:0/3250406261 >> v1:172.21.15.203:6789/0 conn(0x7f10cc06baf0 legacy=0x7f10cc162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.777+0000 7f10c3fff640 1 -- 172.21.15.156:0/3250406261 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc0b5d60 con 0x7f10cc0b40d0 2024-03-29T04:08:58.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.778+0000 7f10c3fff640 1 --2- 172.21.15.156:0/3250406261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc0b40d0 0x7f10cc06b5b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0d2250 tx=0x7f10b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93ab6a7f67485ef server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.778+0000 7f10c9d9b640 1 -- 172.21.15.156:0/3250406261 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4034070 con 0x7f10cc0b40d0 2024-03-29T04:08:58.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.778+0000 7f10c9d9b640 1 -- 172.21.15.156:0/3250406261 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b4030860 con 0x7f10cc0b40d0 2024-03-29T04:08:58.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.778+0000 7f10c9d9b640 1 -- 172.21.15.156:0/3250406261 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4030c30 con 0x7f10cc0b40d0 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.778+0000 7f10d0b11640 1 -- 172.21.15.156:0/3250406261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc0b40d0 msgr2=0x7f10cc06b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.778+0000 7f10d0b11640 1 --2- 172.21.15.156:0/3250406261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc0b40d0 0x7f10cc06b5b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0d2250 tx=0x7f10b402b9a0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.779+0000 7f10d0b11640 1 -- 172.21.15.156:0/3250406261 shutdown_connections 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.779+0000 7f10d0b11640 1 --2- 172.21.15.156:0/3250406261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc0b40d0 0x7f10cc06b5b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.779+0000 7f10d0b11640 1 --2- 172.21.15.156:0/3250406261 >> v2:172.21.15.203:3300/0 conn(0x7f10cc15e720 0x7f10cc0b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.779+0000 7f10d0b11640 1 -- 172.21.15.156:0/3250406261 >> 172.21.15.156:0/3250406261 conn(0x7f10cc09aa40 msgr2=0x7f10cc0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.779+0000 7f10d0b11640 1 -- 172.21.15.156:0/3250406261 shutdown_connections 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.779+0000 7f10d0b11640 1 -- 172.21.15.156:0/3250406261 wait complete. 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.780+0000 7f10d0b11640 1 Processor -- start 2024-03-29T04:08:58.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.780+0000 7f10d0b11640 1 -- start start 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 0x7f10cc161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10cc0b40d0 0x7f10cc1f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f10cc15e720 0x7f10cc15c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc0b6320 con 0x7f10cc06baf0 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc0b2bb0 con 0x7f10cc0b40d0 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f10cc0d1920 con 0x7f10cc15e720 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 0x7f10cc161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 0x7f10cc161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45344/0 (socket says 172.21.15.156:45344) 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 -- 172.21.15.156:0/58014685 learned_addr learned my addr 172.21.15.156:0/58014685 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 -- 172.21.15.156:0/58014685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f10cc15e720 msgr2=0x7f10cc15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f10cc15e720 0x7f10cc15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 -- 172.21.15.156:0/58014685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10cc0b40d0 msgr2=0x7f10cc1f7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10cc0b40d0 0x7f10cc1f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 -- 172.21.15.156:0/58014685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc1608b0 con 0x7f10cc06baf0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10ca59c640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 0x7f10cc161070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e870 tx=0x7f10bc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc017070 con 0x7f10cc06baf0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10bc00c940 con 0x7f10cc06baf0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10cc14d5d0 con 0x7f10cc06baf0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc00cd30 con 0x7f10cc06baf0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.781+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10cc160b50 con 0x7f10cc06baf0 2024-03-29T04:08:58.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.782+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10cc14dc50 con 0x7f10cc06baf0 2024-03-29T04:08:58.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.783+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f10bc01a070 con 0x7f10cc06baf0 2024-03-29T04:08:58.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.784+0000 7f10c2ffd640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f10ac06c530 0x7f10ac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:58.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.784+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f10bc08b4e0 con 0x7f10cc06baf0 2024-03-29T04:08:58.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.784+0000 7f10c3fff640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f10ac06c530 0x7f10ac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:58.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.784+0000 7f10c3fff640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f10ac06c530 0x7f10ac06e9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10b4036000 tx=0x7f10b403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:58.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.788+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f10bc059b90 con 0x7f10cc06baf0 2024-03-29T04:08:58.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.997+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10cc068d20 con 0x7f10cc06baf0 2024-03-29T04:08:58.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:58.998+0000 7f10c2ffd640 1 -- 172.21.15.156:0/58014685 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f10bc059490 con 0x7f10cc06baf0 2024-03-29T04:08:58.978 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:58.978 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:58.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f10ac06c530 msgr2=0x7f10ac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f10ac06c530 0x7f10ac06e9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10b4036000 tx=0x7f10b403b040 comp rx=0 tx=0).stop 2024-03-29T04:08:58.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 msgr2=0x7f10cc161070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:58.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 0x7f10cc161070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e870 tx=0x7f10bc00ed30 comp rx=0 tx=0).stop 2024-03-29T04:08:58.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 shutdown_connections 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f10ac06c530 0x7f10ac06e9f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f10cc15e720 0x7f10cc15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10cc0b40d0 0x7f10cc1f7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 --2- 172.21.15.156:0/58014685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc06baf0 0x7f10cc161070 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.000+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 >> 172.21.15.156:0/58014685 conn(0x7f10cc09aa40 msgr2=0x7f10cc0afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.001+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 shutdown_connections 2024-03-29T04:08:58.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.001+0000 7f10d0b11640 1 -- 172.21.15.156:0/58014685 wait complete. 2024-03-29T04:08:58.991 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-29T04:08:58.991 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:08:59.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.112+0000 7f3aefb9d640 1 Processor -- start 2024-03-29T04:08:59.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.112+0000 7f3aefb9d640 1 -- start start 2024-03-29T04:08:59.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aefb9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae8162340 0x7f3ae80b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aefb9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1bb0 0x7f3ae8162720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aefb9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ae8162c60 0x7f3ae80d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aefb9d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3ae80cd9c0 con 0x7f3ae8162c60 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aefb9d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3ae80b2010 con 0x7f3ae80b1bb0 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aefb9d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae815fcf0 con 0x7f3ae8162340 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aee13a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ae8162c60 0x7f3ae80d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aee13a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ae8162c60 0x7f3ae80d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45358/0 (socket says 172.21.15.156:45358) 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.113+0000 7f3aee13a640 1 -- 172.21.15.156:0/4204924049 learned_addr learned my addr 172.21.15.156:0/4204924049 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aed939640 1 --2- 172.21.15.156:0/4204924049 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae8162340 0x7f3ae80b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aed138640 1 --2- 172.21.15.156:0/4204924049 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1bb0 0x7f3ae8162720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aee13a640 1 -- 172.21.15.156:0/4204924049 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae8162340 msgr2=0x7f3ae80b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aee13a640 1 --2- 172.21.15.156:0/4204924049 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae8162340 0x7f3ae80b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aee13a640 1 -- 172.21.15.156:0/4204924049 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1bb0 msgr2=0x7f3ae8162720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aee13a640 1 --2- 172.21.15.156:0/4204924049 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1bb0 0x7f3ae8162720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aee13a640 1 -- 172.21.15.156:0/4204924049 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae80cdc10 con 0x7f3ae8162c60 2024-03-29T04:08:59.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aed138640 1 --2- 172.21.15.156:0/4204924049 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1bb0 0x7f3ae8162720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:59.100 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.114+0000 7f3aee13a640 1 --2- 172.21.15.156:0/4204924049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162c60 0x7f3ae80d0710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8072fb0 tx=0x7f3ad802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2079f2e0ca89bb01 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.100 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aec937640 1 -- 172.21.15.156:0/4204924049 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad802ca70 con 0x7f3ae8162c60 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aec937640 1 -- 172.21.15.156:0/4204924049 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ad802cc10 con 0x7f3ae8162c60 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aec937640 1 -- 172.21.15.156:0/4204924049 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad80343f0 con 0x7f3ae8162c60 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 -- 172.21.15.156:0/4204924049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162c60 msgr2=0x7f3ae80d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/4204924049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162c60 0x7f3ae80d0710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8072fb0 tx=0x7f3ad802bed0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 -- 172.21.15.156:0/4204924049 shutdown_connections 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/4204924049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162c60 0x7f3ae80d0710 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/4204924049 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1bb0 0x7f3ae8162720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/4204924049 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae8162340 0x7f3ae80b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 -- 172.21.15.156:0/4204924049 >> 172.21.15.156:0/4204924049 conn(0x7f3ae80b3d90 msgr2=0x7f3ae80b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 -- 172.21.15.156:0/4204924049 shutdown_connections 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.115+0000 7f3aefb9d640 1 -- 172.21.15.156:0/4204924049 wait complete. 2024-03-29T04:08:59.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.116+0000 7f3aefb9d640 1 Processor -- start 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.116+0000 7f3aefb9d640 1 -- start start 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aefb9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae80b1bb0 0x7f3ae81575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aefb9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 0x7f3ae806dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aefb9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae8162c60 0x7f3ae8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aefb9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae80d01f0 con 0x7f3ae8162340 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aefb9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae80cff10 con 0x7f3ae8162c60 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aefb9d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae80ce1d0 con 0x7f3ae80b1bb0 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aed138640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 0x7f3ae806dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aed138640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 0x7f3ae806dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45372/0 (socket says 172.21.15.156:45372) 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aed138640 1 -- 172.21.15.156:0/3877165996 learned_addr learned my addr 172.21.15.156:0/3877165996 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aee13a640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae8162c60 0x7f3ae8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aed939640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae80b1bb0 0x7f3ae81575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aed138640 1 -- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae80b1bb0 msgr2=0x7f3ae81575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.117+0000 7f3aed138640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae80b1bb0 0x7f3ae81575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3aed138640 1 -- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae8162c60 msgr2=0x7f3ae8160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3aed138640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae8162c60 0x7f3ae8160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3aed138640 1 -- 172.21.15.156:0/3877165996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae80c7380 con 0x7f3ae8162340 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3aed138640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 0x7f3ae806dfb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00d800 tx=0x7f3adc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc004030 con 0x7f3ae8162340 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3adc0041d0 con 0x7f3ae8162340 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae80af4d0 con 0x7f3ae8162340 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0096a0 con 0x7f3ae8162340 2024-03-29T04:08:59.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.118+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae80afa90 con 0x7f3ae8162340 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.119+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae8068d20 con 0x7f3ae8162340 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.119+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3adc009840 con 0x7f3ae8162340 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.120+0000 7f3ae67fc640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3ad406c530 0x7f3ad406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.120+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3adc08ba30 con 0x7f3ae8162340 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.120+0000 7f3aed939640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3ad406c530 0x7f3ad406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.120+0000 7f3aed939640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3ad406c530 0x7f3ad406e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009a50 tx=0x7f3ae0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.124+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3adc059fc0 con 0x7f3ae8162340 2024-03-29T04:08:59.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.383+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ae80b4940 con 0x7f3ae8162340 2024-03-29T04:08:59.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.384+0000 7f3ae67fc640 1 -- 172.21.15.156:0/3877165996 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f3adc0598c0 con 0x7f3ae8162340 2024-03-29T04:08:59.364 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:08:59.364 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:08:59.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3ad406c530 msgr2=0x7f3ad406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3ad406c530 0x7f3ad406e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009a50 tx=0x7f3ae0008040 comp rx=0 tx=0).stop 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 msgr2=0x7f3ae806dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 0x7f3ae806dfb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00d800 tx=0x7f3adc00dcc0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 shutdown_connections 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3ad406c530 0x7f3ad406e9f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae8162c60 0x7f3ae8160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.386+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ae8162340 0x7f3ae806dfb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.387+0000 7f3aefb9d640 1 --2- 172.21.15.156:0/3877165996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae80b1bb0 0x7f3ae81575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.387+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 >> 172.21.15.156:0/3877165996 conn(0x7f3ae80b3d90 msgr2=0x7f3ae80cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.387+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 shutdown_connections 2024-03-29T04:08:59.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.387+0000 7f3aefb9d640 1 -- 172.21.15.156:0/3877165996 wait complete. 2024-03-29T04:08:59.377 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-29T04:08:59.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-29T04:08:59.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-29T04:08:59.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-29T04:08:59.379 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-29T04:08:59.379 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-29T04:08:59.380 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-29T04:08:59.380 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-29T04:08:59.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.572+0000 7f67ddac8640 1 Processor -- start 2024-03-29T04:08:59.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.572+0000 7f19c9401640 1 Processor -- start 2024-03-29T04:08:59.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.572+0000 7f67ddac8640 1 -- start start 2024-03-29T04:08:59.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.572+0000 7f19c9401640 1 -- start start 2024-03-29T04:08:59.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.573+0000 7f6d1359e640 1 Processor -- start 2024-03-29T04:08:59.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.573+0000 7f6d1359e640 1 -- start start 2024-03-29T04:08:59.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.573+0000 7f67ddac8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 0x7f67d80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.573+0000 7f19c9401640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc050500 0x7f19bc007bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f19c9401640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19bc0512f0 con 0x7f19bc0dfd40 2024-03-29T04:08:59.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67ddac8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f67d806e8b0 0x7f67d815f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f19c9401640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19bc0549d0 con 0x7f19bc0e0120 2024-03-29T04:08:59.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f19c3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f19bc0dfd40 0x7f19bc050be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58972/0 (socket says 172.21.15.156:58972) 2024-03-29T04:08:59.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f19c3fff640 1 -- 172.21.15.156:0/2723554109 learned_addr learned my addr 172.21.15.156:0/2723554109 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f19c9401640 1 -- 172.21.15.156:0/2723554109 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f19bc054cb0 con 0x7f19bc050500 2024-03-29T04:08:59.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67ddac8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d80b55f0 con 0x7f67d815f580 2024-03-29T04:08:59.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67ddac8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f67d80af2c0 con 0x7f67d806e8b0 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67ddac8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f67d80af5a0 con 0x7f67d815b450 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f6d1359e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6d0c04fcf0 0x7f6d0c0382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f6d1359e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c0536a0 con 0x7f6d0c04f870 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f6d1359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c053980 con 0x7f6d0c0e32d0 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f6d1359e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6d0c0e36b0 con 0x7f6d0c04fcf0 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f6d11d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f6d0c04f870 0x7f6d0c037bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58992/0 (socket says 172.21.15.156:58992) 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f6d11d9b640 1 -- 172.21.15.156:0/3593529523 learned_addr learned my addr 172.21.15.156:0/3593529523 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f6d12d9d640 1 --2- 172.21.15.156:0/3593529523 >> v2:172.21.15.203:3300/0 conn(0x7f6d0c04fcf0 0x7f6d0c0382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 568737613 0 0) 0x7f19b0001110 con 0x7f19bc0dfd40 2024-03-29T04:08:59.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19b4003520 con 0x7f19bc0dfd40 2024-03-29T04:08:59.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f6d12d9d640 1 -- 172.21.15.156:0/3593529523 >> v1:172.21.15.203:6789/0 conn(0x7f6d0c0e32d0 legacy=0x7f6d0c052c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f6d12d9d640 1 -- 172.21.15.156:0/3593529523 >> v1:172.21.15.156:6789/0 conn(0x7f6d0c04f870 legacy=0x7f6d0c037bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f6d12d9d640 1 -- 172.21.15.156:0/3593529523 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c053c60 con 0x7f6d0c04fcf0 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67d77fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 0x7f67d80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67d77fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 0x7f67d80b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42510/0 (socket says 172.21.15.156:42510) 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67d77fe640 1 -- 172.21.15.156:0/3098417345 learned_addr learned my addr 172.21.15.156:0/3098417345 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67d6ffd640 1 --2- 172.21.15.156:0/3098417345 >> v2:172.21.15.156:3300/0 conn(0x7f67d806e8b0 0x7f67d815f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.574+0000 7f67d7fff640 1 --1- 172.21.15.156:0/3098417345 >> v1:172.21.15.156:6789/0 conn(0x7f67d815f580 0x7f67d80afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58982/0 (socket says 172.21.15.156:58982) 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f67d67fc640 1 -- 172.21.15.156:0/3098417345 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276368858 0 0) 0x7f67d80b55f0 con 0x7f67d815f580 2024-03-29T04:08:59.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f67d67fc640 1 -- 172.21.15.156:0/3098417345 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67c0003520 con 0x7f67d815f580 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f19c37fe640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc050500 0x7f19bc007bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.575+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439778364 0 0) 0x7f19bc0549d0 con 0x7f19bc0e0120 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19b0001110 con 0x7f19bc0e0120 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2976246792 0 0) 0x7f19b4003520 con 0x7f19bc0dfd40 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc050500 msgr2=0x7f19bc007bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc050500 0x7f19bc007bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 >> v1:172.21.15.203:6789/0 conn(0x7f19bc0e0120 legacy=0x7f19bc008110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc054f90 con 0x7f19bc0dfd40 2024-03-29T04:08:59.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f19b00017c0 con 0x7f19bc0dfd40 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 >> v1:172.21.15.156:6789/0 conn(0x7f19bc0dfd40 legacy=0x7f19bc050be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b4003b70 0x7f19b4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19b4004800 0x7f19b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19b40051c0 0x7f19b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc054cb0 con 0x7f19b4004800 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19b400bf00 con 0x7f19b4003b70 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f19b400c080 con 0x7f19b40051c0 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c8c00640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19b40051c0 0x7f19b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c37fe640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19b4004800 0x7f19b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c3fff640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b4003b70 0x7f19b4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f6d12d9d640 1 --2- 172.21.15.156:0/3593529523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c04fcf0 0x7f6d0c0382c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6d14069d20 tx=0x7f6d140a7100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb718c37e9b9f4e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f67d77fe640 1 -- 172.21.15.156:0/3098417345 >> v2:172.21.15.156:3300/0 conn(0x7f67d806e8b0 msgr2=0x7f67d815f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f67d77fe640 1 --2- 172.21.15.156:0/3098417345 >> v2:172.21.15.156:3300/0 conn(0x7f67d806e8b0 0x7f67d815f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f67d77fe640 1 -- 172.21.15.156:0/3098417345 >> v1:172.21.15.156:6789/0 conn(0x7f67d815f580 legacy=0x7f67d80afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f67d77fe640 1 -- 172.21.15.156:0/3098417345 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d80b58d0 con 0x7f67d815b450 2024-03-29T04:08:59.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.577+0000 7fc932df4640 1 Processor -- start 2024-03-29T04:08:59.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c8c00640 1 -- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b4003b70 msgr2=0x7f19b4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c8c00640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b4003b70 0x7f19b4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c8c00640 1 -- 172.21.15.156:0/2723554109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19b4004800 msgr2=0x7f19b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c8c00640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19b4004800 0x7f19b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.576+0000 7f19c8c00640 1 -- 172.21.15.156:0/2723554109 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc054f90 con 0x7f19b40051c0 2024-03-29T04:08:59.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.577+0000 7f6d1159a640 1 -- 172.21.15.156:0/3593529523 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d140b4070 con 0x7f6d0c04fcf0 2024-03-29T04:08:59.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.577+0000 7f6d1159a640 1 -- 172.21.15.156:0/3593529523 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d140af440 con 0x7f6d0c04fcf0 2024-03-29T04:08:59.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.577+0000 7f6d1159a640 1 -- 172.21.15.156:0/3593529523 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d140af8b0 con 0x7f6d0c04fcf0 2024-03-29T04:08:59.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f19c8c00640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19b40051c0 0x7f19b400b9c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19c4076740 tx=0x7f19b4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911b49e968b83a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 -- 172.21.15.156:0/3593529523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c04fcf0 msgr2=0x7f6d0c0382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 --2- 172.21.15.156:0/3593529523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c04fcf0 0x7f6d0c0382c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6d14069d20 tx=0x7f6d140a7100 comp rx=0 tx=0).stop 2024-03-29T04:08:59.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f67d77fe640 1 --2- 172.21.15.156:0/3098417345 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 0x7f67d80b3580 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009870 tx=0x7f67cc02b830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43b9ff1bc1501c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.577+0000 7fc932df4640 1 -- start start 2024-03-29T04:08:59.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 -- 172.21.15.156:0/3593529523 shutdown_connections 2024-03-29T04:08:59.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 --2- 172.21.15.156:0/3593529523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c04fcf0 0x7f6d0c0382c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 -- 172.21.15.156:0/3593529523 >> 172.21.15.156:0/3593529523 conn(0x7f6d0c009fb0 msgr2=0x7f6d0c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c40b4070 con 0x7f19b40051c0 2024-03-29T04:08:59.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 -- 172.21.15.156:0/3593529523 shutdown_connections 2024-03-29T04:08:59.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc932df4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c15e750 0x7fc92c162d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc932df4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc92c0d14d0 0x7fc92c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc932df4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c15fb40 con 0x7fc92c0cf100 2024-03-29T04:08:59.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc932df4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc92c0cf900 con 0x7fc92c0d14d0 2024-03-29T04:08:59.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc932df4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c0cfbe0 con 0x7fc92c15e750 2024-03-29T04:08:59.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc930b90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c15e750 0x7fc92c162d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f6d1359e640 1 -- 172.21.15.156:0/3593529523 wait complete. 2024-03-29T04:08:59.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc931391640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc92c0d14d0 0x7fc92c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc930b90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c15e750 0x7fc92c162d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42520/0 (socket says 172.21.15.156:42520) 2024-03-29T04:08:59.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7fc930b90640 1 -- 172.21.15.156:0/3257429691 learned_addr learned my addr 172.21.15.156:0/3257429691 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7f67d67fc640 1 -- 172.21.15.156:0/3098417345 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc038070 con 0x7f67d815b450 2024-03-29T04:08:59.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7f67d67fc640 1 -- 172.21.15.156:0/3098417345 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc031510 con 0x7f67d815b450 2024-03-29T04:08:59.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7f67d67fc640 1 -- 172.21.15.156:0/3098417345 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d00019f0 con 0x7f67d815b450 2024-03-29T04:08:59.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7fc92b7fe640 1 -- 172.21.15.156:0/3257429691 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810708226 0 0) 0x7fc92c15fb40 con 0x7fc92c0cf100 2024-03-29T04:08:59.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7fc92b7fe640 1 -- 172.21.15.156:0/3257429691 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc91c003520 con 0x7fc92c0cf100 2024-03-29T04:08:59.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7fc931391640 1 -- 172.21.15.156:0/3257429691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c15e750 msgr2=0x7fc92c162d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7fc931391640 1 --2- 172.21.15.156:0/3257429691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c15e750 0x7fc92c162d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7fc931391640 1 -- 172.21.15.156:0/3257429691 >> v1:172.21.15.203:6789/0 conn(0x7fc92c0cf100 legacy=0x7fc92c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7fc931391640 1 -- 172.21.15.156:0/3257429691 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c0cfec0 con 0x7fc92c0d14d0 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c40a7c00 con 0x7f19b40051c0 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.578+0000 7f19c2ffd640 1 -- 172.21.15.156:0/2723554109 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c40696b0 con 0x7f19b40051c0 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7f19c9401640 1 -- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19b40051c0 msgr2=0x7f19b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7f19c9401640 1 --2- 172.21.15.156:0/2723554109 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19b40051c0 0x7f19b400b9c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19c4076740 tx=0x7f19b4005830 comp rx=0 tx=0).stop 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.579+0000 7f19c8c00640 1 -- 172.21.15.156:0/2723554109 reap_dead start 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f19c9401640 1 -- 172.21.15.156:0/2723554109 shutdown_connections 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f19c9401640 1 -- 172.21.15.156:0/2723554109 >> 172.21.15.156:0/2723554109 conn(0x7f19bc055bc0 msgr2=0x7f19bc055fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f19c9401640 1 -- 172.21.15.156:0/2723554109 shutdown_connections 2024-03-29T04:08:59.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f6d1359e640 1 Processor -- start 2024-03-29T04:08:59.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f6d1359e640 1 -- start start 2024-03-29T04:08:59.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f67ddac8640 1 -- 172.21.15.156:0/3098417345 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 msgr2=0x7f67d80b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7f67ddac8640 1 --2- 172.21.15.156:0/3098417345 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 0x7f67d80b3580 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009870 tx=0x7f67cc02b830 comp rx=0 tx=0).stop 2024-03-29T04:08:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7fc931391640 1 --2- 172.21.15.156:0/3257429691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c0d14d0 0x7fc92c06d880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc924009d90 tx=0x7fc92402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04551e5acf582f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7fc92b7fe640 1 -- 172.21.15.156:0/3257429691 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924039070 con 0x7fc92c0d14d0 2024-03-29T04:08:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.580+0000 7fc92b7fe640 1 -- 172.21.15.156:0/3257429691 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92402cbc0 con 0x7fc92c0d14d0 2024-03-29T04:08:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.581+0000 7fc92b7fe640 1 -- 172.21.15.156:0/3257429691 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924034440 con 0x7fc92c0d14d0 2024-03-29T04:08:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.581+0000 7f19c9401640 1 -- 172.21.15.156:0/2723554109 wait complete. 2024-03-29T04:08:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.581+0000 7f6d1359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c04f870 0x7f6d0c04e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.581+0000 7f6d1359e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d0c04fcf0 0x7f6d0c0d6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.581+0000 7fc932df4640 1 -- 172.21.15.156:0/3257429691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c0d14d0 msgr2=0x7fc92c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.581+0000 7fc932df4640 1 --2- 172.21.15.156:0/3257429691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c0d14d0 0x7fc92c06d880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc924009d90 tx=0x7fc92402be10 comp rx=0 tx=0).stop 2024-03-29T04:08:59.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7f6d1359e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c0e32d0 0x7f6d0c0e4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7fc932df4640 1 -- 172.21.15.156:0/3257429691 shutdown_connections 2024-03-29T04:08:59.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7fc932df4640 1 --2- 172.21.15.156:0/3257429691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c0d14d0 0x7fc92c06d880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7f6d1359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c050160 con 0x7f6d0c04f870 2024-03-29T04:08:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7fc932df4640 1 --2- 172.21.15.156:0/3257429691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c15e750 0x7fc92c162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7f6d1359e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c054220 con 0x7f6d0c0e32d0 2024-03-29T04:08:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7f6d1359e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0c053360 con 0x7f6d0c04fcf0 2024-03-29T04:08:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7fc932df4640 1 -- 172.21.15.156:0/3257429691 >> 172.21.15.156:0/3257429691 conn(0x7fc92c08bfd0 msgr2=0x7fc92c0d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7fc932df4640 1 -- 172.21.15.156:0/3257429691 shutdown_connections 2024-03-29T04:08:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.582+0000 7fc932df4640 1 -- 172.21.15.156:0/3257429691 wait complete. 2024-03-29T04:08:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f67ddac8640 1 -- 172.21.15.156:0/3098417345 shutdown_connections 2024-03-29T04:08:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f67ddac8640 1 --2- 172.21.15.156:0/3098417345 >> v2:172.21.15.156:3300/0 conn(0x7f67d806e8b0 0x7f67d815f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f67ddac8640 1 --2- 172.21.15.156:0/3098417345 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815b450 0x7f67d80b3580 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f67ddac8640 1 -- 172.21.15.156:0/3098417345 >> 172.21.15.156:0/3098417345 conn(0x7f67d80b68b0 msgr2=0x7f67d80b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f6d11d9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d0c04fcf0 0x7f6d0c0d6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f6d1259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c04f870 0x7f6d0c04e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f6d1259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c04f870 0x7f6d0c04e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45408/0 (socket says 172.21.15.156:45408) 2024-03-29T04:08:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.583+0000 7f6d1259c640 1 -- 172.21.15.156:0/1668515551 learned_addr learned my addr 172.21.15.156:0/1668515551 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d12d9d640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c0e32d0 0x7f6d0c0e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7fc932df4640 1 Processor -- start 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7fc932df4640 1 -- start start 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc932df4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c0cf100 0x7fc92c154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc932df4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc92c0d14d0 0x7fc92c161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc932df4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c15e750 0x7fc92c161750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc930b90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c0cf100 0x7fc92c154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc92bfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc92c0d14d0 0x7fc92c161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc92bfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc92c0d14d0 0x7fc92c161210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35162/0 (socket says 172.21.15.156:35162) 2024-03-29T04:08:59.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7fc930b90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c0cf100 0x7fc92c154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42526/0 (socket says 172.21.15.156:42526) 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7fc931391640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c15e750 0x7fc92c161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1259c640 1 -- 172.21.15.156:0/1668515551 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d0c04fcf0 msgr2=0x7f6d0c0d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1259c640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d0c04fcf0 0x7f6d0c0d6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1259c640 1 -- 172.21.15.156:0/1668515551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c0e32d0 msgr2=0x7f6d0c0e4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1259c640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c0e32d0 0x7f6d0c0e4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1259c640 1 -- 172.21.15.156:0/1668515551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c179070 con 0x7f6d0c04f870 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1259c640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c04f870 0x7f6d0c04e0d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800c3c0 tx=0x7f6d0800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08017070 con 0x7f6d0c04f870 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c0e4aa0 con 0x7f6d0c04f870 2024-03-29T04:08:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c0e5060 con 0x7f6d0c04f870 2024-03-29T04:08:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.584+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d080052f0 con 0x7f6d0c04f870 2024-03-29T04:08:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.585+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08003830 con 0x7f6d0c04f870 2024-03-29T04:08:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6cf8001030 con 0x7f6d0c04f870 2024-03-29T04:08:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.586+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6d0801a070 con 0x7f6d0c04f870 2024-03-29T04:08:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7fc931391640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c15e750 0x7fc92c161750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45418/0 (socket says 172.21.15.156:45418) 2024-03-29T04:08:59.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7f6d12d9d640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c0e32d0 0x7f6d0c0e4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:59.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7fc92bfff640 1 -- 172.21.15.156:0/1469978088 learned_addr learned my addr 172.21.15.156:0/1469978088 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7fc932df4640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c0cf560 con 0x7fc92c15e750 2024-03-29T04:08:59.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7f6d02ffd640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6cf406c580 0x7f6cf406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7f67ddac8640 1 -- 172.21.15.156:0/3098417345 shutdown_connections 2024-03-29T04:08:59.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7f67ddac8640 1 -- 172.21.15.156:0/3098417345 wait complete. 2024-03-29T04:08:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7fc932df4640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c0d0480 con 0x7fc92c0d14d0 2024-03-29T04:08:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7fc932df4640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c0cf6e0 con 0x7fc92c0cf100 2024-03-29T04:08:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6d0808a780 con 0x7f6d0c04f870 2024-03-29T04:08:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.587+0000 7f6d02ffd640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f6cf40747a0 0x7f6cf4076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7f6d12d9d640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f6cf40747a0 0x7f6cf4076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc931391640 1 -- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c0cf100 msgr2=0x7fc92c154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc931391640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc92c0cf100 0x7fc92c154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc931391640 1 -- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc92c0d14d0 msgr2=0x7fc92c161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc931391640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc92c0d14d0 0x7fc92c161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc931391640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c0ca720 con 0x7fc92c15e750 2024-03-29T04:08:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 --> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d080052f0 con 0x7f6cf40747a0 2024-03-29T04:08:59.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc931391640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c15e750 0x7fc92c161750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc92c08d0c0 tx=0x7fc924004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6d0808ab40 con 0x7f6d0c04f870 2024-03-29T04:08:59.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.589+0000 7f67ddac8640 1 Processor -- start 2024-03-29T04:08:59.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.589+0000 7f6d12d9d640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f6cf40747a0 0x7f6cf4076bc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.589+0000 7f67ddac8640 1 -- start start 2024-03-29T04:08:59.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924039050 con 0x7fc92c15e750 2024-03-29T04:08:59.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.588+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc924045070 con 0x7fc92c15e750 2024-03-29T04:08:59.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.589+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924039050 con 0x7fc92c15e750 2024-03-29T04:08:59.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.590+0000 7fc932df4640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c0b3dd0 con 0x7fc92c15e750 2024-03-29T04:08:59.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.589+0000 7f6d11d9b640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6cf406c580 0x7f6cf406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.590+0000 7f6d11d9b640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6cf406c580 0x7f6cf406ea40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c039dc0 tx=0x7f6d04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.590+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== osd.2 v2:172.21.15.156:6824/196737254 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d080052f0 con 0x7f6cf40747a0 2024-03-29T04:08:59.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.590+0000 7fd0dca89640 1 Processor -- start 2024-03-29T04:08:59.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.590+0000 7fd0dca89640 1 -- start start 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67ddac8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67d806e8b0 0x7f67d81f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67ddac8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67d815b450 0x7f67d8151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67ddac8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815f580 0x7f67d80b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67ddac8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f67d815feb0 con 0x7f67d806e8b0 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67ddac8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67d815fd30 con 0x7f67d815b450 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67ddac8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f67d806ecf0 con 0x7f67d815f580 2024-03-29T04:08:59.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67d806e8b0 0x7f67d81f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67d806e8b0 0x7f67d81f01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45424/0 (socket says 172.21.15.156:45424) 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 -- 172.21.15.156:0/2693136627 learned_addr learned my addr 172.21.15.156:0/2693136627 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d6ffd640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67d815b450 0x7f67d8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 -- 172.21.15.156:0/2693136627 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815f580 msgr2=0x7f67d80b40d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67d815f580 0x7f67d80b40d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 -- 172.21.15.156:0/2693136627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67d815b450 msgr2=0x7f67d8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67d815b450 0x7f67d8151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7f67d77fe640 1 -- 172.21.15.156:0/2693136627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d815cc60 con 0x7f67d806e8b0 2024-03-29T04:08:59.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0dca89640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd0d00df210 0x7fd0d003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0dca89640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00e2d70 0x7fd0d00e31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0dca89640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd0d00533d0 0x7fd0d0053860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0dca89640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd0d00e3710 con 0x7fd0d00533d0 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0dca89640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd0d0054630 con 0x7fd0d00df210 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0dca89640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d0054910 con 0x7fd0d00e2d70 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fd0d6ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00e2d70 0x7fd0d00e31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd0d00533d0 0x7fd0d0053860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd0d00533d0 0x7fd0d0053860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45438/0 (socket says 172.21.15.156:45438) 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 -- 172.21.15.156:0/3689757392 learned_addr learned my addr 172.21.15.156:0/3689757392 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d6ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00e2d70 0x7fd0d00e31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42548/0 (socket says 172.21.15.156:42548) 2024-03-29T04:08:59.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.591+0000 7fc932df4640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c0b4390 con 0x7fc92c15e750 2024-03-29T04:08:59.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d77fe640 1 --2- 172.21.15.156:0/3689757392 >> v2:172.21.15.203:3300/0 conn(0x7fd0d00df210 0x7fd0d003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7f67d77fe640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67d806e8b0 0x7f67d81f01d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67cc02b870 tx=0x7f67cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc92c0b4a50 con 0x7fc92c15e750 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 -- 172.21.15.156:0/3689757392 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00e2d70 msgr2=0x7fd0d00e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 --2- 172.21.15.156:0/3689757392 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00e2d70 0x7fd0d00e31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 -- 172.21.15.156:0/3689757392 >> v2:172.21.15.203:3300/0 conn(0x7fd0d00df210 msgr2=0x7fd0d003a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 --2- 172.21.15.156:0/3689757392 >> v2:172.21.15.203:3300/0 conn(0x7fd0d00df210 0x7fd0d003a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.592+0000 7fd0d7fff640 1 -- 172.21.15.156:0/3689757392 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d0054bf0 con 0x7fd0d00533d0 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc924040450 con 0x7fc92c15e750 2024-03-29T04:08:59.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7f19c9401640 1 Processor -- start 2024-03-29T04:08:59.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc042070 con 0x7f67d806e8b0 2024-03-29T04:08:59.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc03d870 con 0x7f67d806e8b0 2024-03-29T04:08:59.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc038040 con 0x7f67d806e8b0 2024-03-29T04:08:59.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d815cf40 con 0x7f67d806e8b0 2024-03-29T04:08:59.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d815d500 con 0x7f67d806e8b0 2024-03-29T04:08:59.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7fc9297fa640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc91006c960 0x7fc91006ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.593+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc92402c9c0 con 0x7fc92c15e750 2024-03-29T04:08:59.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7fc9297fa640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fc910074b00 0x7fc910076f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc92402cbc0 con 0x7fc910074b00 2024-03-29T04:08:59.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7f19c9401640 1 -- start start 2024-03-29T04:08:59.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7fc930b90640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc91006c960 0x7fc91006ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc9240b2060 con 0x7fc92c15e750 2024-03-29T04:08:59.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7fc930b90640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc91006c960 0x7fc91006ee20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc918009b30 tx=0x7fc918009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.594+0000 7f214fd52640 1 Processor -- start 2024-03-29T04:08:59.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7f214fd52640 1 -- start start 2024-03-29T04:08:59.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7fd0d7fff640 1 --2- 172.21.15.156:0/3689757392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00533d0 0x7fd0d0053860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000b470 tx=0x7fd0cc02bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d843059cd19242 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7fd0d67fc640 1 -- 172.21.15.156:0/3689757392 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc039070 con 0x7fd0d00533d0 2024-03-29T04:08:59.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f67b8001030 con 0x7f67d806e8b0 2024-03-29T04:08:59.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7fd0d67fc640 1 -- 172.21.15.156:0/3689757392 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0cc02c9d0 con 0x7fd0d00533d0 2024-03-29T04:08:59.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7fd0d67fc640 1 -- 172.21.15.156:0/3689757392 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc02cda0 con 0x7fd0d00533d0 2024-03-29T04:08:59.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.595+0000 7fc92bfff640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fc910074b00 0x7fc910076f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7fc92bfff640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fc910074b00 0x7fc910076f20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f67cc045070 con 0x7f67d806e8b0 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7f214fd52640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f214806dd40 0x7f214806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7f214daee640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f214806d960 0x7f2148072110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:59006/0 (socket says 172.21.15.156:59006) 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214d2ed640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f21480717c0 0x7f21480ad030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:54758/0 (socket says 172.21.15.156:54758) 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7fd0dca89640 1 -- 172.21.15.156:0/3689757392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00533d0 msgr2=0x7fd0d0053860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7fd0dca89640 1 --2- 172.21.15.156:0/3689757392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00533d0 0x7fd0d0053860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000b470 tx=0x7fd0cc02bc20 comp rx=0 tx=0).stop 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7fd0dca89640 1 -- 172.21.15.156:0/3689757392 shutdown_connections 2024-03-29T04:08:59.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7fd0dca89640 1 --2- 172.21.15.156:0/3689757392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00533d0 0x7fd0d0053860 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.596+0000 7fd0dca89640 1 --2- 172.21.15.156:0/3689757392 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00e2d70 0x7fd0d00e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7fd0dca89640 1 --2- 172.21.15.156:0/3689757392 >> v2:172.21.15.203:3300/0 conn(0x7fd0d00df210 0x7fd0d003a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7fd0dca89640 1 -- 172.21.15.156:0/3689757392 >> 172.21.15.156:0/3689757392 conn(0x7fd0d000a3f0 msgr2=0x7fd0d000a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214fd52640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21480b34b0 con 0x7f214806d960 2024-03-29T04:08:59.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214daee640 1 -- 172.21.15.156:0/896891503 learned_addr learned my addr 172.21.15.156:0/896891503 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214e2ef640 1 --2- 172.21.15.156:0/896891503 >> v2:172.21.15.203:3300/0 conn(0x7f214806dd40 0x7f214806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== osd.5 v2:172.21.15.203:6808/2129623277 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc92402cbc0 con 0x7fc910074b00 2024-03-29T04:08:59.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21480b3790 con 0x7f21480717c0 2024-03-29T04:08:59.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f21480b3a70 con 0x7f214806dd40 2024-03-29T04:08:59.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7fd0dca89640 1 -- 172.21.15.156:0/3689757392 shutdown_connections 2024-03-29T04:08:59.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7fd0dca89640 1 -- 172.21.15.156:0/3689757392 wait complete. 2024-03-29T04:08:59.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214caec640 1 -- 172.21.15.156:0/896891503 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355755358 0 0) 0x7f21480b3790 con 0x7f21480717c0 2024-03-29T04:08:59.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214caec640 1 -- 172.21.15.156:0/896891503 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2134003520 con 0x7f21480717c0 2024-03-29T04:08:59.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214e2ef640 1 -- 172.21.15.156:0/896891503 >> v1:172.21.15.203:6789/0 conn(0x7f21480717c0 legacy=0x7f21480ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214e2ef640 1 -- 172.21.15.156:0/896891503 >> v1:172.21.15.156:6789/0 conn(0x7f214806d960 legacy=0x7f2148072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f214e2ef640 1 -- 172.21.15.156:0/896891503 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21480b3d50 con 0x7f214806dd40 2024-03-29T04:08:59.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.597+0000 7f67bffff640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f67c406c580 0x7f67c406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.598+0000 7f67d6ffd640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f67c406c580 0x7f67c406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.598+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f67cc0ad080 con 0x7f67d806e8b0 2024-03-29T04:08:59.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.598+0000 7f67bffff640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f67c4074720 0x7f67c4076b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 --> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67c40771f0 con 0x7f67c4074720 2024-03-29T04:08:59.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f214e2ef640 1 --2- 172.21.15.156:0/896891503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214806e1b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f214000c2e0 tx=0x7f214002d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a45a885c55f5345e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.598+0000 7fc609b1c640 1 Processor -- start 2024-03-29T04:08:59.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.598+0000 7fc609b1c640 1 -- start start 2024-03-29T04:08:59.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f67cc0b4050 con 0x7f67d806e8b0 2024-03-29T04:08:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f67d7fff640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f67c4074720 0x7f67c4076b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f214caec640 1 -- 172.21.15.156:0/896891503 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140039070 con 0x7f214806dd40 2024-03-29T04:08:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f214caec640 1 -- 172.21.15.156:0/896891503 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f214000a4e0 con 0x7f214806dd40 2024-03-29T04:08:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f214caec640 1 -- 172.21.15.156:0/896891503 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214000a8b0 con 0x7f214806dd40 2024-03-29T04:08:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7f67d7fff640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f67c4074720 0x7f67c4076b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7f67d6ffd640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f67c406c580 0x7f67c406ea40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f67d8158fb0 tx=0x7f67c8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc609b1c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6040717c0 0x7fc6040ad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc609b1c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6040b34b0 con 0x7fc60406f840 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc609b1c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6040b3790 con 0x7fc60406d960 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc609b1c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc6040b3a70 con 0x7fc6040717c0 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc6037fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc60406d960 0x7fc604072110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:54770/0 (socket says 172.21.15.156:54770) 2024-03-29T04:08:59.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc6037fe640 1 -- 172.21.15.156:0/2316673228 learned_addr learned my addr 172.21.15.156:0/2316673228 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc602ffd640 1 --2- 172.21.15.156:0/2316673228 >> v2:172.21.15.203:3300/0 conn(0x7fc6040717c0 0x7fc6040ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc602ffd640 1 -- 172.21.15.156:0/2316673228 >> v1:172.21.15.203:6789/0 conn(0x7fc60406d960 legacy=0x7fc604072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc602ffd640 1 -- 172.21.15.156:0/2316673228 >> v1:172.21.15.156:6789/0 conn(0x7fc60406f840 legacy=0x7fc60406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fc602ffd640 1 -- 172.21.15.156:0/2316673228 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6040b3d50 con 0x7fc6040717c0 2024-03-29T04:08:59.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 msgr2=0x7f214806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7f214fd52640 1 --2- 172.21.15.156:0/896891503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214806e1b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f214000c2e0 tx=0x7f214002d430 comp rx=0 tx=0).stop 2024-03-29T04:08:59.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 shutdown_connections 2024-03-29T04:08:59.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7f214fd52640 1 --2- 172.21.15.156:0/896891503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214806e1b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 >> 172.21.15.156:0/896891503 conn(0x7f21480b1d50 msgr2=0x7f21480b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== osd.3 v2:172.21.15.156:6816/3904521534 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67c40771f0 con 0x7f67c4074720 2024-03-29T04:08:59.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 shutdown_connections 2024-03-29T04:08:59.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.598+0000 7fd0dca89640 1 Processor -- start 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.599+0000 7fd0dca89640 1 -- start start 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0dca89640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00533d0 0x7fd0d00d85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0dca89640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00df210 0x7fd0d0036740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0d77fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00533d0 0x7fd0d00d85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0d77fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00533d0 0x7fd0d00d85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42552/0 (socket says 172.21.15.156:42552) 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0d77fe640 1 -- 172.21.15.156:0/1534223008 learned_addr learned my addr 172.21.15.156:0/1534223008 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0d6ffd640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00df210 0x7fd0d0036740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0dca89640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0d00e2d70 0x7fd0d0036c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d00df5f0 con 0x7fd0d00df210 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d00065b0 con 0x7fd0d00e2d70 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.600+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d00e3a50 con 0x7fd0d00533d0 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0d77fe640 1 -- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0d00e2d70 msgr2=0x7fd0d0036c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0d77fe640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0d00e2d70 0x7fd0d0036c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0d77fe640 1 -- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00df210 msgr2=0x7fd0d0036740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0d77fe640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0d00df210 0x7fd0d0036740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0d77fe640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d00e06a0 con 0x7fd0d00533d0 2024-03-29T04:08:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f214fd52640 1 -- 172.21.15.156:0/896891503 wait complete. 2024-03-29T04:08:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0d77fe640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00533d0 0x7fd0d00d85d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400d830 tx=0x7fd0c400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fc602ffd640 1 --2- 172.21.15.156:0/2316673228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6040717c0 0x7fc6040ad030 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4011fc0 tx=0x7fc5f4034180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4226d7b37f9c7ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f19c9401640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b40051c0 0x7f19bc033dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c400b6c0 con 0x7fd0d00533d0 2024-03-29T04:08:59.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f19c9401640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc035570 0x7f19bc0359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f19c9401640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19bc0377d0 0x7f19bc037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f19c9401640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc039e00 con 0x7f19bc0377d0 2024-03-29T04:08:59.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f19c9401640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc055550 con 0x7f19b40051c0 2024-03-29T04:08:59.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7f19c9401640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f19bc039f80 con 0x7f19bc035570 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fc6027fc640 1 -- 172.21.15.156:0/2316673228 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f40110d0 con 0x7fc6040717c0 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.601+0000 7fc6027fc640 1 -- 172.21.15.156:0/2316673228 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f4011270 con 0x7fc6040717c0 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.602+0000 7fc6027fc640 1 -- 172.21.15.156:0/2316673228 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4011640 con 0x7fc6040717c0 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.602+0000 7fc609b1c640 1 -- 172.21.15.156:0/2316673228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6040717c0 msgr2=0x7fc6040ad030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.602+0000 7fc609b1c640 1 --2- 172.21.15.156:0/2316673228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6040717c0 0x7fc6040ad030 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4011fc0 tx=0x7fc5f4034180 comp rx=0 tx=0).stop 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7fc609b1c640 1 -- 172.21.15.156:0/2316673228 shutdown_connections 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7fc609b1c640 1 --2- 172.21.15.156:0/2316673228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6040717c0 0x7fc6040ad030 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7fc609b1c640 1 -- 172.21.15.156:0/2316673228 >> 172.21.15.156:0/2316673228 conn(0x7fc6040b1d50 msgr2=0x7fc6040b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7fc609b1c640 1 -- 172.21.15.156:0/2316673228 shutdown_connections 2024-03-29T04:08:59.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.602+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d00e0980 con 0x7fd0d00533d0 2024-03-29T04:08:59.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.602+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d00e0f40 con 0x7fd0d00533d0 2024-03-29T04:08:59.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c4019070 con 0x7fd0d00533d0 2024-03-29T04:08:59.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7fc609b1c640 1 -- 172.21.15.156:0/2316673228 wait complete. 2024-03-29T04:08:59.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b40051c0 0x7f19bc033dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b40051c0 0x7f19bc033dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35216/0 (socket says 172.21.15.156:35216) 2024-03-29T04:08:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c3fff640 1 -- 172.21.15.156:0/3200821000 learned_addr learned my addr 172.21.15.156:0/3200821000 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f214fd52640 1 Processor -- start 2024-03-29T04:08:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c37fe640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc035570 0x7f19bc0359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c3fff640 1 -- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc035570 msgr2=0x7f19bc0359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c3fff640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc035570 0x7f19bc0359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c3fff640 1 -- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19bc0377d0 msgr2=0x7f19bc037c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T04:08:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f214fd52640 1 -- start start 2024-03-29T04:08:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.603+0000 7f19c8c00640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19bc0377d0 0x7f19bc037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.604+0000 7f19c3fff640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19bc0377d0 0x7f19bc037c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.604+0000 7f19c3fff640 1 -- 172.21.15.156:0/3200821000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc038530 con 0x7f19b40051c0 2024-03-29T04:08:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.604+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c400ba60 con 0x7fd0d00533d0 2024-03-29T04:08:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.604+0000 7fc609b1c640 1 Processor -- start 2024-03-29T04:08:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7fc609b1c640 1 -- start start 2024-03-29T04:08:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214fd52640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214806d960 0x7f214809a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214fd52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214809c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214daee640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214806d960 0x7f214809a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214fd52640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21480717c0 0x7f21481f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214d2ed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214809c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214d2ed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214809c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35228/0 (socket says 172.21.15.156:35228) 2024-03-29T04:08:59.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.604+0000 7f19c37fe640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19bc035570 0x7f19bc0359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:59.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214d2ed640 1 -- 172.21.15.156:0/2394776832 learned_addr learned my addr 172.21.15.156:0/2394776832 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f214806f840 con 0x7f21480717c0 2024-03-29T04:08:59.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21480ad740 con 0x7f214806dd40 2024-03-29T04:08:59.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214e2ef640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21480717c0 0x7f21481f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f214806f9c0 con 0x7f214806d960 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.605+0000 7f19c8c00640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19bc0377d0 0x7f19bc037c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc609b1c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc60406d960 0x7fc604153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc609b1c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc60406f840 0x7fc604151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc609b1c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6040717c0 0x7fc60414b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc609b1c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc6040ad570 con 0x7fc6040717c0 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc609b1c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc60406e450 con 0x7fc60406d960 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc609b1c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc6040ad6f0 con 0x7fc60406f840 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc603fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6040717c0 0x7fc60414b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc603fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6040717c0 0x7fc60414b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45462/0 (socket says 172.21.15.156:45462) 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fc603fff640 1 -- 172.21.15.156:0/1137023533 learned_addr learned my addr 172.21.15.156:0/1137023533 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd0c400bc40 con 0x7fd0d00533d0 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7f214d2ed640 1 -- 172.21.15.156:0/2394776832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214806d960 msgr2=0x7f214809a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7f214d2ed640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214806d960 0x7f214809a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7f214d2ed640 1 -- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21480717c0 msgr2=0x7f21481f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7f214d2ed640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21480717c0 0x7f21481f0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7f214d2ed640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21481f13e0 con 0x7f214806dd40 2024-03-29T04:08:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd0b8001030 con 0x7fd0d00533d0 2024-03-29T04:08:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc602ffd640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc60406f840 0x7fc604151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.606+0000 7f19c3fff640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b40051c0 0x7f19bc033dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19b0007310 tx=0x7f19b00077d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7f214d2ed640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214809c3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f213800c3c0 tx=0x7f2138001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc6037fe640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc60406d960 0x7fc604153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fd0c3fff640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd0bc06c600 0x7fd0bc06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b000fa80 con 0x7f19b40051c0 2024-03-29T04:08:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19b0018070 con 0x7f19b40051c0 2024-03-29T04:08:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b000fe60 con 0x7f19b40051c0 2024-03-29T04:08:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fd0d6ffd640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd0bc06c600 0x7fd0bc06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc603fff640 1 -- 172.21.15.156:0/1137023533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc60406f840 msgr2=0x7fc604151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc603fff640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc60406f840 0x7fc604151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc603fff640 1 -- 172.21.15.156:0/1137023533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc60406d960 msgr2=0x7fc604153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc603fff640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc60406d960 0x7fc604153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc603fff640 1 -- 172.21.15.156:0/1137023533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc604152480 con 0x7fc6040717c0 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7fc603fff640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6040717c0 0x7fc60414b690 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00c730 tx=0x7fc5fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc038810 con 0x7f19b40051c0 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.607+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc0381c0 con 0x7f19b40051c0 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.608+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21380074e0 con 0x7f214806dd40 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21380047e0 con 0x7f214806dd40 2024-03-29T04:08:59.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138004bd0 con 0x7f214806dd40 2024-03-29T04:08:59.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19bc038a50 con 0x7f19b40051c0 2024-03-29T04:08:59.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.608+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc016020 con 0x7fc6040717c0 2024-03-29T04:08:59.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.608+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc006d50 con 0x7fc6040717c0 2024-03-29T04:08:59.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.608+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc003bb0 con 0x7fc6040717c0 2024-03-29T04:08:59.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.608+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc604152760 con 0x7fc6040717c0 2024-03-29T04:08:59.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.608+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc604152d20 con 0x7fc6040717c0 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f19b003a5e0 con 0x7f19b40051c0 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2148154d60 con 0x7f214806dd40 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd0c408ba00 con 0x7fd0d00533d0 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7fd0c3fff640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fd0bc0747a0 0x7fd0bc076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0c4019070 con 0x7fd0bc0747a0 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.609+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd0c4092050 con 0x7fd0d00533d0 2024-03-29T04:08:59.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7fd0d6ffd640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd0bc06c600 0x7fd0bc06eac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8069d20 tx=0x7fd0d807d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2148155320 con 0x7f214806dd40 2024-03-29T04:08:59.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7fd0d7fff640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fd0bc0747a0 0x7fd0bc076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc5e4001030 con 0x7fc6040717c0 2024-03-29T04:08:59.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc5fc0050d0 con 0x7fc6040717c0 2024-03-29T04:08:59.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7fc5ebfff640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc5f006c600 0x7fc5f006eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7fd0d7fff640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fd0bc0747a0 0x7fd0bc076bc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc5fc08b0e0 con 0x7fc6040717c0 2024-03-29T04:08:59.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.611+0000 7f19c0ff9640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f19a806c530 0x7f19a806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f19b0057b90 con 0x7f19b40051c0 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c0ff9640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f19a80752e0 0x7f19a8077780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 --> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19a8077e50 con 0x7f19a80752e0 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f19b0056520 con 0x7f19b40051c0 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c37fe640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f19a806c530 0x7f19a806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c8c00640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f19a80752e0 0x7f19a8077780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c8c00640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f19a80752e0 0x7f19a8077780 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7f19c37fe640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f19a806c530 0x7f19a806e9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0dd610 tx=0x7f19b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7fc5ebfff640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fc5f0074880 0x7fc5f0076ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 --> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5fc012e50 con 0x7fc5f0074880 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.612+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc5fc092050 con 0x7fc6040717c0 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.613+0000 7fc602ffd640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fc5f0074880 0x7fc5f0076ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.613+0000 7fc6037fe640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc5f006c600 0x7fc5f006eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.613+0000 7fc6037fe640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc5f006c600 0x7fc5f006eac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc6041f36a0 tx=0x7fc5f8007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.613+0000 7fc602ffd640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fc5f0074880 0x7fc5f0076ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.613+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2148155700 con 0x7f214806dd40 2024-03-29T04:08:59.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.614+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f213801a020 con 0x7f214806dd40 2024-03-29T04:08:59.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.614+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== osd.7 v2:172.21.15.203:6817/3097526350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f19a8077e50 con 0x7f19a80752e0 2024-03-29T04:08:59.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.614+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== osd.4 v2:172.21.15.203:6816/3017904752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd0c4019070 con 0x7fd0bc0747a0 2024-03-29T04:08:59.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.614+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== osd.0 v2:172.21.15.156:6808/1070206615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc5fc012e50 con 0x7fc5f0074880 2024-03-29T04:08:59.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.614+0000 7f213e7fc640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f212c06c600 0x7f212c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.614+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f213808ae60 con 0x7f214806dd40 2024-03-29T04:08:59.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.615+0000 7f213e7fc640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f212c074800 0x7f212c076c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.615+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2138015e70 con 0x7f212c074800 2024-03-29T04:08:59.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.615+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2138055000 con 0x7f214806dd40 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.615+0000 7f214e2ef640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f212c074800 0x7f212c076c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.616+0000 7f214e2ef640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f212c074800 0x7f212c076c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.617+0000 7f214daee640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f212c06c600 0x7f212c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.617+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== osd.1 v2:172.21.15.156:6800/413252124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2138015e70 con 0x7f212c074800 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.619+0000 7f1a6a855640 1 Processor -- start 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.619+0000 7f1a6a855640 1 -- start start 2024-03-29T04:08:59.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.620+0000 7f1a6a855640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1a640cef30 0x7f1a640cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.620+0000 7f1a6a855640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a640b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.620+0000 7f1a6a855640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a640b0a80 con 0x7f1a64162a90 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.620+0000 7f1a6a855640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1a64163e80 con 0x7f1a640cef30 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.620+0000 7f1a6a855640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1a640b2eb0 con 0x7f1a640b2500 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1a640cef30 0x7f1a640cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1a640cef30 0x7f1a640cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45474/0 (socket says 172.21.15.156:45474) 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 -- 172.21.15.156:0/257206498 learned_addr learned my addr 172.21.15.156:0/257206498 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.619+0000 7f214daee640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f212c06c600 0x7f212c06eac0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2144000f60 tx=0x7f2144009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a68df2640 1 --2- 172.21.15.156:0/257206498 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a640b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 -- 172.21.15.156:0/257206498 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 msgr2=0x7f1a640b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 --2- 172.21.15.156:0/257206498 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a640b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 -- 172.21.15.156:0/257206498 >> v1:172.21.15.156:6789/0 conn(0x7f1a64162a90 legacy=0x7f1a640b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.621+0000 7f1a637fe640 1 -- 172.21.15.156:0/257206498 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a640b0d60 con 0x7f1a640cef30 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.623+0000 7f1a637fe640 1 --2- 172.21.15.156:0/257206498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a640cef30 0x7f1a640cf3b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009e80 tx=0x7f1a5c02bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31c8b09c168c1bf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.625+0000 7f1a62ffd640 1 -- 172.21.15.156:0/257206498 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c02c840 con 0x7f1a640cef30 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.625+0000 7f1a62ffd640 1 -- 172.21.15.156:0/257206498 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c02c9e0 con 0x7f1a640cef30 2024-03-29T04:08:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.626+0000 7f1a6a855640 1 -- 172.21.15.156:0/257206498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a640cef30 msgr2=0x7f1a640cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.626+0000 7f1a6a855640 1 --2- 172.21.15.156:0/257206498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a640cef30 0x7f1a640cf3b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009e80 tx=0x7f1a5c02bd20 comp rx=0 tx=0).stop 2024-03-29T04:08:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.627+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 --> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6cf8002dd0 con 0x7f6cf40747a0 2024-03-29T04:08:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.627+0000 7f6d02ffd640 1 -- 172.21.15.156:0/1668515551 <== osd.2 v2:172.21.15.156:6824/196737254 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6cf8002dd0 con 0x7f6cf40747a0 2024-03-29T04:08:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.627+0000 7f1a6a855640 1 -- 172.21.15.156:0/257206498 shutdown_connections 2024-03-29T04:08:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.627+0000 7f1a6a855640 1 --2- 172.21.15.156:0/257206498 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a640b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.627+0000 7f1a6a855640 1 --2- 172.21.15.156:0/257206498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a640cef30 0x7f1a640cf3b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.627+0000 7f1a6a855640 1 -- 172.21.15.156:0/257206498 >> 172.21.15.156:0/257206498 conn(0x7f1a64070050 msgr2=0x7f1a6406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.628+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f6cf40747a0 msgr2=0x7f6cf4076bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.628+0000 7f6d1359e640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f6cf40747a0 0x7f6cf4076bc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.628+0000 7f1a6a855640 1 -- 172.21.15.156:0/257206498 shutdown_connections 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.628+0000 7f1a6a855640 1 -- 172.21.15.156:0/257206498 wait complete. 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.630+0000 7f1a6a855640 1 Processor -- start 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.630+0000 7f1a6a855640 1 -- start start 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7f1a6a855640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a6414bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7f1a6a855640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a640cef30 0x7f1a6414d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7f1a6a855640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a64162a90 0x7f1a6415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7f1a6a855640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1a640b1600 con 0x7f1a64162a90 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6cf406c580 msgr2=0x7f6cf406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d1359e640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6cf406c580 0x7f6cf406ea40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c039dc0 tx=0x7f6d04009040 comp rx=0 tx=0).stop 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c04f870 msgr2=0x7f6d0c04e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d1359e640 1 --2- 172.21.15.156:0/1668515551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d0c04f870 0x7f6d0c04e0d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800c3c0 tx=0x7f6d0800c880 comp rx=0 tx=0).stop 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d12d9d640 1 -- 172.21.15.156:0/1668515551 reap_dead start 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7f1a6a855640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1a640b1320 con 0x7f1a640cef30 2024-03-29T04:08:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7f1a6a855640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1a640b0730 con 0x7f1a640b2500 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 shutdown_connections 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 >> 172.21.15.156:0/1668515551 conn(0x7f6d0c009fb0 msgr2=0x7f6d0c0e05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f1a68df2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a64162a90 0x7f1a6415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f1a68df2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a64162a90 0x7f1a6415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45476/0 (socket says 172.21.15.156:45476) 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f1a68df2640 1 -- 172.21.15.156:0/2750358602 learned_addr learned my addr 172.21.15.156:0/2750358602 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f1a63fff640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a6414bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7f1a637fe640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a640cef30 0x7f1a6414d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 --> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc92c068d20 con 0x7fc910074b00 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.631+0000 7fc9297fa640 1 -- 172.21.15.156:0/1469978088 <== osd.5 v2:172.21.15.203:6808/2129623277 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc92c068d20 con 0x7fc910074b00 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fc910074b00 msgr2=0x7fc910076f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc916ffd640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fc910074b00 0x7fc910076f20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc91006c960 msgr2=0x7fc91006ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc916ffd640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc91006c960 0x7fc91006ee20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc918009b30 tx=0x7fc918009240 comp rx=0 tx=0).stop 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c15e750 msgr2=0x7fc92c161750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc916ffd640 1 --2- 172.21.15.156:0/1469978088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc92c15e750 0x7fc92c161750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc92c08d0c0 tx=0x7fc924004a40 comp rx=0 tx=0).stop 2024-03-29T04:08:59.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.632+0000 7fc931391640 1 -- 172.21.15.156:0/1469978088 reap_dead start 2024-03-29T04:08:59.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 shutdown_connections 2024-03-29T04:08:59.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f6d1359e640 1 -- 172.21.15.156:0/1668515551 wait complete. 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f1a68df2640 1 -- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 msgr2=0x7f1a6414bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f1a68df2640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a640b2500 0x7f1a6414bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f1a68df2640 1 -- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a640cef30 msgr2=0x7f1a6414d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f1a68df2640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a640cef30 0x7f1a6414d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f1a68df2640 1 -- 172.21.15.156:0/2750358602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6414db20 con 0x7f1a64162a90 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.633+0000 7f1a68df2640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a64162a90 0x7f1a6415cf90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400d800 tx=0x7f1a5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.634+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54018020 con 0x7f1a64162a90 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.634+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a540096c0 con 0x7f1a64162a90 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.635+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54009ab0 con 0x7f1a64162a90 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.635+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6414c820 con 0x7f1a64162a90 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.635+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6414cde0 con 0x7f1a64162a90 2024-03-29T04:08:59.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.637+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1a6406e6f0 con 0x7f1a64162a90 2024-03-29T04:08:59.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.634+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 shutdown_connections 2024-03-29T04:08:59.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.638+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1a5401b020 con 0x7f1a64162a90 2024-03-29T04:08:59.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.638+0000 7f1a60ff9640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a5006c600 0x7f1a5006eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1a5408b7f0 con 0x7f1a64162a90 2024-03-29T04:08:59.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f1a60ff9640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f1a500747a0 0x7f1a50076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 --> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a54008f30 con 0x7f1a500747a0 2024-03-29T04:08:59.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 --> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f67b8002dd0 con 0x7f67c4074720 2024-03-29T04:08:59.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f1a63fff640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a5006c600 0x7f1a5006eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.634+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 >> 172.21.15.156:0/1469978088 conn(0x7fc92c08bfd0 msgr2=0x7fc92c0b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1a54092050 con 0x7f1a64162a90 2024-03-29T04:08:59.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.639+0000 7f1a637fe640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f1a500747a0 0x7f1a50076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.640+0000 7f67bffff640 1 -- 172.21.15.156:0/2693136627 <== osd.3 v2:172.21.15.156:6816/3904521534 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f67b8002dd0 con 0x7f67c4074720 2024-03-29T04:08:59.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.640+0000 7f1a637fe640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f1a500747a0 0x7f1a50076bc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.640+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 shutdown_connections 2024-03-29T04:08:59.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.640+0000 7f1a63fff640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a5006c600 0x7f1a5006eac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a58000f60 tx=0x7f1a58009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.640+0000 7fc916ffd640 1 -- 172.21.15.156:0/1469978088 wait complete. 2024-03-29T04:08:59.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== osd.6 v2:172.21.15.203:6800/505413700 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a54008f30 con 0x7f1a500747a0 2024-03-29T04:08:59.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f67c4074720 msgr2=0x7f67c4076b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67ddac8640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f67c4074720 0x7f67c4076b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f67c406c580 msgr2=0x7f67c406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67ddac8640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f67c406c580 0x7f67c406ea40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f67d8158fb0 tx=0x7f67c8009190 comp rx=0 tx=0).stop 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67d806e8b0 msgr2=0x7f67d81f01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67ddac8640 1 --2- 172.21.15.156:0/2693136627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67d806e8b0 0x7f67d81f01d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67cc02b870 tx=0x7f67cc0046c0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.641+0000 7f67d7fff640 1 -- 172.21.15.156:0/2693136627 reap_dead start 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.642+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 shutdown_connections 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.642+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 >> 172.21.15.156:0/2693136627 conn(0x7f67d80b68b0 msgr2=0x7f67d8159870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.642+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 shutdown_connections 2024-03-29T04:08:59.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.642+0000 7f67ddac8640 1 -- 172.21.15.156:0/2693136627 wait complete. 2024-03-29T04:08:59.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.646+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 --> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc5e4002d50 con 0x7fc5f0074880 2024-03-29T04:08:59.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.646+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 --> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f19bc0057d0 con 0x7f19a80752e0 2024-03-29T04:08:59.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.646+0000 7fc5ebfff640 1 -- 172.21.15.156:0/1137023533 <== osd.0 v2:172.21.15.156:6808/1070206615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc5e4002d50 con 0x7fc5f0074880 2024-03-29T04:08:59.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.647+0000 7f19c0ff9640 1 -- 172.21.15.156:0/3200821000 <== osd.7 v2:172.21.15.203:6817/3097526350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f19bc0057d0 con 0x7f19a80752e0 2024-03-29T04:08:59.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.647+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f19a80752e0 msgr2=0x7f19a8077780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.647+0000 7f19c9401640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f19a80752e0 0x7f19a8077780 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.653 INFO:teuthology.orchestra.run.smithi156.stdout:55834574854 2024-03-29T04:08:59.653 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T04:08:59.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.650+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f19a806c530 msgr2=0x7f19a806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f19a806c530 0x7f19a806e9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0dd610 tx=0x7f19b8008040 comp rx=0 tx=0).stop 2024-03-29T04:08:59.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fc5f0074880 msgr2=0x7fc5f0076ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fc5f0074880 0x7fc5f0076ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b40051c0 msgr2=0x7f19bc033dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 --2- 172.21.15.156:0/3200821000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19b40051c0 0x7f19bc033dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19b0007310 tx=0x7f19b00077d0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c8c00640 1 -- 172.21.15.156:0/3200821000 reap_dead start 2024-03-29T04:08:59.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 shutdown_connections 2024-03-29T04:08:59.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 >> 172.21.15.156:0/3200821000 conn(0x7f19bc055bc0 msgr2=0x7f19bc008a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 shutdown_connections 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc5f006c600 msgr2=0x7fc5f006eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc5f006c600 0x7fc5f006eac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc6041f36a0 tx=0x7fc5f8007e80 comp rx=0 tx=0).stop 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6040717c0 msgr2=0x7fc60414b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 --2- 172.21.15.156:0/1137023533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6040717c0 0x7fc60414b690 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00c730 tx=0x7fc5fc00cbf0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc603fff640 1 -- 172.21.15.156:0/1137023533 reap_dead start 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 shutdown_connections 2024-03-29T04:08:59.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 >> 172.21.15.156:0/1137023533 conn(0x7fc6040b1d50 msgr2=0x7fc604071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 shutdown_connections 2024-03-29T04:08:59.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7fc609b1c640 1 -- 172.21.15.156:0/1137023533 wait complete. 2024-03-29T04:08:59.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.651+0000 7f19c9401640 1 -- 172.21.15.156:0/3200821000 wait complete. 2024-03-29T04:08:59.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.652+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 --> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd0b8002dd0 con 0x7fd0bc0747a0 2024-03-29T04:08:59.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.653+0000 7fd0c3fff640 1 -- 172.21.15.156:0/1534223008 <== osd.4 v2:172.21.15.203:6816/3017904752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd0b8002dd0 con 0x7fd0bc0747a0 2024-03-29T04:08:59.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fd0bc0747a0 msgr2=0x7fd0bc076bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0dca89640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fd0bc0747a0 0x7fd0bc076bc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd0bc06c600 msgr2=0x7fd0bc06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0dca89640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd0bc06c600 0x7fd0bc06eac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8069d20 tx=0x7fd0d807d040 comp rx=0 tx=0).stop 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00533d0 msgr2=0x7fd0d00d85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0dca89640 1 --2- 172.21.15.156:0/1534223008 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0d00533d0 0x7fd0d00d85d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400d830 tx=0x7fd0c400dcf0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7fd0d7fff640 1 -- 172.21.15.156:0/1534223008 reap_dead start 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.654+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 --> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2148068610 con 0x7f212c074800 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.655+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 shutdown_connections 2024-03-29T04:08:59.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.655+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 >> 172.21.15.156:0/1534223008 conn(0x7fd0d000a3f0 msgr2=0x7fd0d00494c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.655+0000 7f213e7fc640 1 -- 172.21.15.156:0/2394776832 <== osd.1 v2:172.21.15.156:6800/413252124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2148068610 con 0x7f212c074800 2024-03-29T04:08:59.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.655+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 shutdown_connections 2024-03-29T04:08:59.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.655+0000 7fd0dca89640 1 -- 172.21.15.156:0/1534223008 wait complete. 2024-03-29T04:08:59.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.656+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f212c074800 msgr2=0x7f212c076c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.656+0000 7f214fd52640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f212c074800 0x7f212c076c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.656+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f212c06c600 msgr2=0x7f212c06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.656+0000 7f214fd52640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f212c06c600 0x7f212c06eac0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2144000f60 tx=0x7f2144009040 comp rx=0 tx=0).stop 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.657+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 msgr2=0x7f214809c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.657+0000 7f214fd52640 1 --2- 172.21.15.156:0/2394776832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f214806dd40 0x7f214809c3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f213800c3c0 tx=0x7f2138001e70 comp rx=0 tx=0).stop 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.657+0000 7f214e2ef640 1 -- 172.21.15.156:0/2394776832 reap_dead start 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.657+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 shutdown_connections 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.657+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 >> 172.21.15.156:0/2394776832 conn(0x7f21480b1d50 msgr2=0x7f21480ae6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.658+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 shutdown_connections 2024-03-29T04:08:59.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.658+0000 7f214fd52640 1 -- 172.21.15.156:0/2394776832 wait complete. 2024-03-29T04:08:59.663 INFO:teuthology.orchestra.run.smithi156.stdout:60129542151 2024-03-29T04:08:59.663 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-29T04:08:59.664 INFO:teuthology.orchestra.run.smithi156.stdout:55834574854 2024-03-29T04:08:59.664 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-29T04:08:59.665 INFO:teuthology.orchestra.run.smithi156.stdout:55834574854 2024-03-29T04:08:59.665 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T04:08:59.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.670+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 --> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1a64068d20 con 0x7f1a500747a0 2024-03-29T04:08:59.666 INFO:teuthology.orchestra.run.smithi156.stdout:60129542150 2024-03-29T04:08:59.666 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-29T04:08:59.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a60ff9640 1 -- 172.21.15.156:0/2750358602 <== osd.6 v2:172.21.15.203:6800/505413700 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1a64068d20 con 0x7f1a500747a0 2024-03-29T04:08:59.668 INFO:teuthology.orchestra.run.smithi156.stdout:60129542151 2024-03-29T04:08:59.668 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f1a500747a0 msgr2=0x7f1a50076bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f1a500747a0 0x7f1a50076bc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a5006c600 msgr2=0x7f1a5006eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a5006c600 0x7f1a5006eac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a58000f60 tx=0x7f1a58009040 comp rx=0 tx=0).stop 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a64162a90 msgr2=0x7f1a6415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 --2- 172.21.15.156:0/2750358602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a64162a90 0x7f1a6415cf90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400d800 tx=0x7f1a5400dcc0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a68df2640 1 -- 172.21.15.156:0/2750358602 reap_dead start 2024-03-29T04:08:59.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 shutdown_connections 2024-03-29T04:08:59.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.671+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 >> 172.21.15.156:0/2750358602 conn(0x7f1a64070050 msgr2=0x7f1a64161090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.672+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 shutdown_connections 2024-03-29T04:08:59.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.672+0000 7f1a6a855640 1 -- 172.21.15.156:0/2750358602 wait complete. 2024-03-29T04:08:59.670 INFO:teuthology.orchestra.run.smithi156.stdout:55834574854 2024-03-29T04:08:59.671 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T04:08:59.671 INFO:teuthology.orchestra.run.smithi156.stdout:60129542150 2024-03-29T04:08:59.672 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-29T04:08:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.893+0000 7f508d6f7640 1 Processor -- start 2024-03-29T04:08:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.893+0000 7f508d6f7640 1 -- start start 2024-03-29T04:08:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.894+0000 7f0b73fff640 1 Processor -- start 2024-03-29T04:08:59.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.895+0000 7f508d6f7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5088155820 0x7f50880b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.895+0000 7f0b73fff640 1 -- start start 2024-03-29T04:08:59.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.896+0000 7f0b73fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0b6c0e12b0 0x7f0b6c0534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.896+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c054b50 0x7f0b6c053a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.896+0000 7f0b73fff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6c009bd0 con 0x7f0b6c0e4d30 2024-03-29T04:08:59.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b73fff640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0b6c054fe0 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b73fff640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c0552c0 con 0x7f0b6c054b50 2024-03-29T04:08:59.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b727fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c054b50 0x7f0b6c053a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b737fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0b6c0e4d30 0x7f0b6c0e1690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:54774/0 (socket says 172.21.15.156:54774) 2024-03-29T04:08:59.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b737fe640 1 -- 172.21.15.156:0/3603767093 learned_addr learned my addr 172.21.15.156:0/3603767093 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.896+0000 7f508d6f7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508806e640 con 0x7f508815b900 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.896+0000 7f508d6f7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50880b4980 con 0x7f508806ee00 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.896+0000 7f508d6f7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f508806e970 con 0x7f5088155820 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f50877fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f508815b900 0x7f508815c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:59042/0 (socket says 172.21.15.156:59042) 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f50877fe640 1 -- 172.21.15.156:0/4050316745 learned_addr learned my addr 172.21.15.156:0/4050316745 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f5086ffd640 1 -- 172.21.15.156:0/4050316745 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897823743 0 0) 0x7f508806e640 con 0x7f508815b900 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f5087fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5088155820 0x7f50880b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f5086ffd640 1 -- 172.21.15.156:0/4050316745 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5070003520 con 0x7f508815b900 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f5086ffd640 1 -- 172.21.15.156:0/4050316745 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3321081094 0 0) 0x7f5070003520 con 0x7f508815b900 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.898+0000 7f5087fff640 1 -- 172.21.15.156:0/4050316745 >> v1:172.21.15.203:6789/0 conn(0x7f508806ee00 legacy=0x7f50880b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.898+0000 7f5087fff640 1 -- 172.21.15.156:0/4050316745 >> v1:172.21.15.156:6789/0 conn(0x7f508815b900 legacy=0x7f508815c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.898+0000 7f5087fff640 1 -- 172.21.15.156:0/4050316745 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50880b4c60 con 0x7f5088155820 2024-03-29T04:08:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b71ffb640 1 -- 172.21.15.156:0/3603767093 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710712193 0 0) 0x7f0b6c009bd0 con 0x7f0b6c0e4d30 2024-03-29T04:08:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b71ffb640 1 -- 172.21.15.156:0/3603767093 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b5c003520 con 0x7f0b6c0e4d30 2024-03-29T04:08:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b727fc640 1 -- 172.21.15.156:0/3603767093 >> v2:172.21.15.203:3300/0 conn(0x7f0b6c0e12b0 msgr2=0x7f0b6c0534f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T04:08:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b727fc640 1 --2- 172.21.15.156:0/3603767093 >> v2:172.21.15.203:3300/0 conn(0x7f0b6c0e12b0 0x7f0b6c0534f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b727fc640 1 -- 172.21.15.156:0/3603767093 >> v1:172.21.15.203:6789/0 conn(0x7f0b6c0e4d30 legacy=0x7f0b6c0e1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.897+0000 7f0b727fc640 1 -- 172.21.15.156:0/3603767093 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c0555a0 con 0x7f0b6c054b50 2024-03-29T04:08:59.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.898+0000 7f5087fff640 1 --2- 172.21.15.156:0/4050316745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088155820 0x7f50880b4150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f50780097a0 tx=0x7f507802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=192c15e755f426e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.899+0000 7f5086ffd640 1 -- 172.21.15.156:0/4050316745 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078037070 con 0x7f5088155820 2024-03-29T04:08:59.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.899+0000 7f0b727fc640 1 --2- 172.21.15.156:0/3603767093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c054b50 0x7f0b6c053a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b640097a0 tx=0x7f0b640340e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16c48aba27cde26b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.899+0000 7f5086ffd640 1 -- 172.21.15.156:0/4050316745 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5078030510 con 0x7f5088155820 2024-03-29T04:08:59.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.899+0000 7f5086ffd640 1 -- 172.21.15.156:0/4050316745 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c001650 con 0x7f5088155820 2024-03-29T04:08:59.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.899+0000 7f0b71ffb640 1 -- 172.21.15.156:0/3603767093 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64041070 con 0x7f0b6c054b50 2024-03-29T04:08:59.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.899+0000 7f0b71ffb640 1 -- 172.21.15.156:0/3603767093 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b6403c440 con 0x7f0b6c054b50 2024-03-29T04:08:59.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b73fff640 1 -- 172.21.15.156:0/3603767093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c054b50 msgr2=0x7f0b6c053a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f112dbab640 1 Processor -- start 2024-03-29T04:08:59.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b71ffb640 1 -- 172.21.15.156:0/3603767093 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64041070 con 0x7f0b6c054b50 2024-03-29T04:08:59.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f112dbab640 1 -- start start 2024-03-29T04:08:59.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b73fff640 1 --2- 172.21.15.156:0/3603767093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c054b50 0x7f0b6c053a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b640097a0 tx=0x7f0b640340e0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f508d6f7640 1 -- 172.21.15.156:0/4050316745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088155820 msgr2=0x7f50880b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b73fff640 1 -- 172.21.15.156:0/3603767093 shutdown_connections 2024-03-29T04:08:59.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b73fff640 1 --2- 172.21.15.156:0/3603767093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c054b50 0x7f0b6c053a30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b73fff640 1 --2- 172.21.15.156:0/3603767093 >> v2:172.21.15.203:3300/0 conn(0x7f0b6c0e12b0 0x7f0b6c0534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f0b73fff640 1 -- 172.21.15.156:0/3603767093 >> 172.21.15.156:0/3603767093 conn(0x7f0b6c020040 msgr2=0x7f0b6c056d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.900+0000 7f508d6f7640 1 --2- 172.21.15.156:0/4050316745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088155820 0x7f50880b4150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f50780097a0 tx=0x7f507802b790 comp rx=0 tx=0).stop 2024-03-29T04:08:59.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f508d6f7640 1 -- 172.21.15.156:0/4050316745 shutdown_connections 2024-03-29T04:08:59.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f508d6f7640 1 --2- 172.21.15.156:0/4050316745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088155820 0x7f50880b4150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f508d6f7640 1 -- 172.21.15.156:0/4050316745 >> 172.21.15.156:0/4050316745 conn(0x7f508806c6c0 msgr2=0x7f508806caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f508d6f7640 1 -- 172.21.15.156:0/4050316745 shutdown_connections 2024-03-29T04:08:59.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f508d6f7640 1 -- 172.21.15.156:0/4050316745 wait complete. 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f112dbab640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f11200e31c0 0x7f1120006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f112dbab640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1120007190 0x7f11200e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f112dbab640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11200076f0 con 0x7f11200e7360 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f112dbab640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f11200e77b0 con 0x7f1120007190 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.901+0000 7f112dbab640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1120057670 con 0x7f11200e31c0 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f112cba9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f11200e31c0 0x7f1120006c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f112cba9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f11200e31c0 0x7f1120006c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35268/0 (socket says 172.21.15.156:35268) 2024-03-29T04:08:59.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f112cba9640 1 -- 172.21.15.156:0/2395819430 learned_addr learned my addr 172.21.15.156:0/2395819430 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f112cba9640 1 -- 172.21.15.156:0/2395819430 >> v2:172.21.15.156:3300/0 conn(0x7f1120007190 msgr2=0x7f11200e6e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T04:08:59.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f6cc99ec640 1 Processor -- start 2024-03-29T04:08:59.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f0b73fff640 1 -- 172.21.15.156:0/3603767093 shutdown_connections 2024-03-29T04:08:59.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f1127fff640 1 --2- 172.21.15.156:0/2395819430 >> v2:172.21.15.156:3300/0 conn(0x7f1120007190 0x7f11200e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f112cba9640 1 --2- 172.21.15.156:0/2395819430 >> v2:172.21.15.156:3300/0 conn(0x7f1120007190 0x7f11200e6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f112cba9640 1 -- 172.21.15.156:0/2395819430 >> v1:172.21.15.156:6789/0 conn(0x7f11200e7360 legacy=0x7f11200537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f7fb7fff640 1 Processor -- start 2024-03-29T04:08:59.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f7fb7fff640 1 -- start start 2024-03-29T04:08:59.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.903+0000 7f6cc99ec640 1 -- start start 2024-03-29T04:08:59.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.903+0000 7f112cba9640 1 -- 172.21.15.156:0/2395819430 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11200079d0 con 0x7f11200e31c0 2024-03-29T04:08:59.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.902+0000 7f0b73fff640 1 -- 172.21.15.156:0/3603767093 wait complete. 2024-03-29T04:08:59.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.903+0000 7f7fb7fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb0053c60 0x7f7fb0038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb7fff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb00541d0 con 0x7f7fb000c090 2024-03-29T04:08:59.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb7fff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb0005f20 con 0x7f7fb0033ee0 2024-03-29T04:08:59.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb7fff640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb0006200 con 0x7f7fb0053c60 2024-03-29T04:08:59.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f1127fff640 1 --2- 172.21.15.156:0/2395819430 >> v2:172.21.15.156:3300/0 conn(0x7f1120007190 0x7f11200e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:59.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb6ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7fb000c090 0x7f7fb00defa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:59060/0 (socket says 172.21.15.156:59060) 2024-03-29T04:08:59.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb6ffd640 1 -- 172.21.15.156:0/595546308 learned_addr learned my addr 172.21.15.156:0/595546308 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f0b73fff640 1 Processor -- start 2024-03-29T04:08:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f508d6f7640 1 Processor -- start 2024-03-29T04:08:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f0b73fff640 1 -- start start 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581817286 0 0) 0x7f7fb0005f20 con 0x7f7fb0033ee0 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa0003520 con 0x7f7fb0033ee0 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744949517 0 0) 0x7f7fb00541d0 con 0x7f7fb000c090 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb0005f20 con 0x7f7fb000c090 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7fdce3959640 1 Processor -- start 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb0053c60 0x7f7fb0038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3767254472 0 0) 0x7f7fa0003520 con 0x7f7fb0033ee0 2024-03-29T04:08:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb0053c60 msgr2=0x7f7fb0038070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb0053c60 0x7f7fb0038070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 >> v1:172.21.15.156:6789/0 conn(0x7f7fb000c090 legacy=0x7f7fb00defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb00064e0 con 0x7f7fb0033ee0 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f7fb80696b0 con 0x7f7fb0033ee0 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 >> v1:172.21.15.203:6789/0 conn(0x7f7fb0033ee0 legacy=0x7f7fb004fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa0003b40 0x7f7fa0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00051c0 0x7f7fa000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb0006200 con 0x7f7fa0004800 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa000bf00 con 0x7f7fa00051c0 2024-03-29T04:08:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa000c100 con 0x7f7fa0003b40 2024-03-29T04:08:59.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7fdce3959640 1 -- start start 2024-03-29T04:08:59.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb0053c60 0x7f7fb0038070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:59.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb6ffd640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa0003b40 0x7f7fa0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb77fe640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00051c0 0x7f7fa000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b6c0e12b0 0x7f0b6c04a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c0e4d30 0x7f0b6c038100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f112cba9640 1 --2- 172.21.15.156:0/2395819430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e31c0 0x7f1120006c50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11200085d0 tx=0x7f111c02ba60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91107da45f11df65 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b6c0d7530 0x7f0b6c0dbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f0b73fff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c052c80 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f0b73fff640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c052b00 con 0x7f0b6c0d7530 2024-03-29T04:08:59.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f0b73fff640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c0576a0 con 0x7f0b6c0e4d30 2024-03-29T04:08:59.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f0b727fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c0e4d30 0x7f0b6c038100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.904+0000 7f508d6f7640 1 -- start start 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508d6f7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f508806ee00 0x7f508814fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508d6f7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5088155820 0x7f5088152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508d6f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 0x7f50880b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508d6f7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5088155c00 con 0x7f508806ee00 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508d6f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50880b4690 con 0x7f508815b900 2024-03-29T04:08:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508d6f7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f508815d020 con 0x7f5088155820 2024-03-29T04:08:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f0b727fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c0e4d30 0x7f0b6c038100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42628/0 (socket says 172.21.15.156:42628) 2024-03-29T04:08:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f0b727fc640 1 -- 172.21.15.156:0/2432854198 learned_addr learned my addr 172.21.15.156:0/2432854198 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f0b72ffd640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b6c0e12b0 0x7f0b6c04a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508cef6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 0x7f50880b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508cef6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 0x7f50880b0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35300/0 (socket says 172.21.15.156:35300) 2024-03-29T04:08:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f508cef6640 1 -- 172.21.15.156:0/17132800 learned_addr learned my addr 172.21.15.156:0/17132800 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f11277fe640 1 -- 172.21.15.156:0/2395819430 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c02d820 con 0x7f11200e31c0 2024-03-29T04:08:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f11277fe640 1 -- 172.21.15.156:0/2395819430 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111c02d9c0 con 0x7f11200e31c0 2024-03-29T04:08:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce3959640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc0d0280 0x7fdcdc0d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce3959640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdcdc0b3d80 0x7fdcdc06d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f0b737fe640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b6c0d7530 0x7f0b6c0dbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce3959640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcdc0d0ba0 con 0x7fdcdc06dba0 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc99ec640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6cbc0dfef0 0x7f6cbc0085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc99ec640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6cbc0e42d0 0x7f6cbc006ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc99ec640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cbc056590 con 0x7f6cbc0e02d0 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc99ec640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6cbc0094b0 con 0x7f6cbc0e42d0 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc99ec640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6cbc056870 con 0x7f6cbc0dfef0 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc89ea640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6cbc0dfef0 0x7f6cbc0085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f6cc3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6cbc0e02d0 0x7f6cbc0e3bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:54808/0 (socket says 172.21.15.156:54808) 2024-03-29T04:08:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f6cc3fff640 1 -- 172.21.15.156:0/1885188247 learned_addr learned my addr 172.21.15.156:0/1885188247 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce3959640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdcdc06e300 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce3959640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdcdc06e020 con 0x7fdcdc0d0280 2024-03-29T04:08:59.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f5087fff640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f508806ee00 0x7f508814fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce0ef4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdcdc0b3d80 0x7fdcdc06d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce0ef4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdcdc0b3d80 0x7fdcdc06d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35302/0 (socket says 172.21.15.156:35302) 2024-03-29T04:08:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce0ef4640 1 -- 172.21.15.156:0/3790842239 learned_addr learned my addr 172.21.15.156:0/3790842239 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7fdce16f5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc0d0280 0x7fdcdc0d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f6cc91eb640 1 --2- 172.21.15.156:0/1885188247 >> v2:172.21.15.156:3300/0 conn(0x7f6cbc0e42d0 0x7f6cbc006ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508cef6640 1 -- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5088155820 msgr2=0x7f5088152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508cef6640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5088155820 0x7f5088152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508cef6640 1 -- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f508806ee00 msgr2=0x7f508814fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f11277fe640 1 -- 172.21.15.156:0/2395819430 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c02dd90 con 0x7f11200e31c0 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb6ffd640 1 -- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00051c0 msgr2=0x7f7fa000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb6ffd640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00051c0 0x7f7fa000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb6ffd640 1 -- 172.21.15.156:0/595546308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fa0004800 msgr2=0x7f7fa0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.905+0000 7f7fb6ffd640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7fdce0ef4640 1 -- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc0d0280 msgr2=0x7fdcdc0d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7fdce0ef4640 1 --2- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc0d0280 0x7fdcdc0d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7fdce0ef4640 1 -- 172.21.15.156:0/3790842239 >> v1:172.21.15.203:6789/0 conn(0x7fdcdc06dba0 legacy=0x7fdcdc166500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7fdce0ef4640 1 -- 172.21.15.156:0/3790842239 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc071c50 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.906+0000 7f7fb6ffd640 1 -- 172.21.15.156:0/595546308 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa0003520 con 0x7f7fa0003b40 2024-03-29T04:08:59.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f0b72ffd640 1 -- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c0e4d30 msgr2=0x7f0b6c038100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f0b72ffd640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c0e4d30 0x7f0b6c038100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508cef6640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f508806ee00 0x7f508814fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508cef6640 1 -- 172.21.15.156:0/17132800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5088154430 con 0x7f508815b900 2024-03-29T04:08:59.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f0b72ffd640 1 -- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b6c0d7530 msgr2=0x7f0b6c0dbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f0b72ffd640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b6c0d7530 0x7f0b6c0dbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f0b72ffd640 1 -- 172.21.15.156:0/2432854198 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c04af60 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f112dbab640 1 -- 172.21.15.156:0/2395819430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e31c0 msgr2=0x7f1120006c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f112dbab640 1 --2- 172.21.15.156:0/2395819430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e31c0 0x7f1120006c50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11200085d0 tx=0x7f111c02ba60 comp rx=0 tx=0).stop 2024-03-29T04:08:59.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508cef6640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 0x7f50880b0cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f508000c780 tx=0x7f508000cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f6cc37fe640 1 -- 172.21.15.156:0/1885188247 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866144671 0 0) 0x7f6cbc056590 con 0x7f6cbc0e02d0 2024-03-29T04:08:59.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f6cc37fe640 1 -- 172.21.15.156:0/1885188247 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb4003520 con 0x7f6cbc0e02d0 2024-03-29T04:08:59.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f0b72ffd640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b6c0e12b0 0x7f0b6c04a8a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009fd0 tx=0x7f0b6800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080017070 con 0x7f508815b900 2024-03-29T04:08:59.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f112dbab640 1 -- 172.21.15.156:0/2395819430 shutdown_connections 2024-03-29T04:08:59.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f6ea87f4640 1 Processor -- start 2024-03-29T04:08:59.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f112dbab640 1 --2- 172.21.15.156:0/2395819430 >> v2:172.21.15.156:3300/0 conn(0x7f1120007190 0x7f11200e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f112dbab640 1 --2- 172.21.15.156:0/2395819430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e31c0 0x7f1120006c50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f112dbab640 1 -- 172.21.15.156:0/2395819430 >> 172.21.15.156:0/2395819430 conn(0x7f112003aa30 msgr2=0x7f112003ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5088154710 con 0x7f508815b900 2024-03-29T04:08:59.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5088154cd0 con 0x7f508815b900 2024-03-29T04:08:59.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f6cc89ea640 1 -- 172.21.15.156:0/1885188247 >> v2:172.21.15.156:3300/0 conn(0x7f6cbc0e42d0 msgr2=0x7f6cbc006ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f6cc37fe640 1 -- 172.21.15.156:0/1885188247 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2693982444 0 0) 0x7f6cb4003520 con 0x7f6cbc0e02d0 2024-03-29T04:08:59.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.907+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5080003d80 con 0x7f508815b900 2024-03-29T04:08:59.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7fdce0ef4640 1 --2- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc06d660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009870 tx=0x7fdcd402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18faac9ffecc2186 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f7fb6ffd640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa0003b40 0x7f7fa0003f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80018e0 tx=0x7f7fa0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c65e39698ea0f0f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68015020 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b68011830 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f112dbab640 1 -- 172.21.15.156:0/2395819430 shutdown_connections 2024-03-29T04:08:59.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f112dbab640 1 -- 172.21.15.156:0/2395819430 wait complete. 2024-03-29T04:08:59.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50800052b0 con 0x7f508815b900 2024-03-29T04:08:59.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68011c80 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6c04b240 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6c04b800 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5068005290 con 0x7f508815b900 2024-03-29T04:08:59.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f6ea87f4640 1 -- start start 2024-03-29T04:08:59.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f6cc89ea640 1 --2- 172.21.15.156:0/1885188247 >> v2:172.21.15.156:3300/0 conn(0x7f6cbc0e42d0 0x7f6cbc006ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f6cc89ea640 1 -- 172.21.15.156:0/1885188247 >> v1:172.21.15.203:6789/0 conn(0x7f6cbc0e02d0 legacy=0x7f6cbc0e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.908+0000 7f6cc89ea640 1 -- 172.21.15.156:0/1885188247 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc008e60 con 0x7f6cbc0dfef0 2024-03-29T04:08:59.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6cc91eb640 1 --2- 172.21.15.156:0/1885188247 >> v2:172.21.15.156:3300/0 conn(0x7f6cbc0e42d0 0x7f6cbc006ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:59.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea87f4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0072220 0x7f6ea016d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea87f4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea006e5b0 con 0x7f6ea0162970 2024-03-29T04:08:59.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea87f4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea006e870 con 0x7f6ea00cda30 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea87f4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea006eb50 con 0x7f6ea0072220 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea5d8f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6ea00cda30 0x7f6ea0071b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:54826/0 (socket says 172.21.15.156:54826) 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea5d8f640 1 -- 172.21.15.156:0/3077909098 learned_addr learned my addr 172.21.15.156:0/3077909098 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f6ea6d91640 1 --2- 172.21.15.156:0/3077909098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0072220 0x7f6ea016d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7fdcd3fff640 1 -- 172.21.15.156:0/3790842239 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4035020 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7fdcd3fff640 1 -- 172.21.15.156:0/3790842239 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd4031640 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7fdcd3fff640 1 -- 172.21.15.156:0/3790842239 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd402d040 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.909+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8001500 con 0x7f7fa0003b40 2024-03-29T04:08:59.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f112dbab640 1 Processor -- start 2024-03-29T04:08:59.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7fdce3959640 1 -- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0b3d80 msgr2=0x7fdcdc06d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7fdce3959640 1 --2- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc06d660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009870 tx=0x7fdcd402b8f0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f6ea6d91640 1 -- 172.21.15.156:0/3077909098 >> v1:172.21.15.203:6789/0 conn(0x7f6ea00cda30 legacy=0x7f6ea0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f6ea6d91640 1 -- 172.21.15.156:0/3077909098 >> v1:172.21.15.156:6789/0 conn(0x7f6ea0162970 legacy=0x7f6ea00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f6ea6d91640 1 -- 172.21.15.156:0/3077909098 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea006ee30 con 0x7f6ea0072220 2024-03-29T04:08:59.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f112dbab640 1 -- start start 2024-03-29T04:08:59.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa8004aa0 con 0x7f7fa0003b40 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.910+0000 7f7fb5ffb640 1 -- 172.21.15.156:0/595546308 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8007070 con 0x7f7fa0003b40 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f7fb7fff640 1 -- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa0003b40 msgr2=0x7f7fa0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/595546308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa0003b40 0x7f7fa0003f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80018e0 tx=0x7f7fa0005830 comp rx=0 tx=0).stop 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f7fb77fe640 1 -- 172.21.15.156:0/595546308 reap_dead start 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b6c0057e0 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f7fb7fff640 1 -- 172.21.15.156:0/595546308 shutdown_connections 2024-03-29T04:08:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f7fb7fff640 1 -- 172.21.15.156:0/595546308 >> 172.21.15.156:0/595546308 conn(0x7f7fb0009fa0 msgr2=0x7f7fb000a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7fdce3959640 1 -- 172.21.15.156:0/3790842239 shutdown_connections 2024-03-29T04:08:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7fdce3959640 1 --2- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc06d660 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7fdce3959640 1 --2- 172.21.15.156:0/3790842239 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc0d0280 0x7fdcdc0d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7fdce3959640 1 -- 172.21.15.156:0/3790842239 >> 172.21.15.156:0/3790842239 conn(0x7fdcdc0b1f30 msgr2=0x7fdcdc0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f6ea6d91640 1 --2- 172.21.15.156:0/3077909098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0072220 0x7f6ea016d630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009dc0 tx=0x7f6e9802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32ccaca16b814297 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f6cc89ea640 1 --2- 172.21.15.156:0/1885188247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc0085d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc038840 tx=0x7f6cb8034910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ae9b523df485a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f6cc37fe640 1 -- 172.21.15.156:0/1885188247 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0001670 con 0x7f6cbc0dfef0 2024-03-29T04:08:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f6cc37fe640 1 -- 172.21.15.156:0/1885188247 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb8040070 con 0x7f6cbc0dfef0 2024-03-29T04:08:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f6cc37fe640 1 -- 172.21.15.156:0/1885188247 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb803a810 con 0x7f6cbc0dfef0 2024-03-29T04:08:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f6ea558e640 1 -- 172.21.15.156:0/3077909098 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98039070 con 0x7f6ea0072220 2024-03-29T04:08:59.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.911+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0b68018020 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f112dbab640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120007190 0x7f11200e5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f7fb7fff640 1 -- 172.21.15.156:0/595546308 shutdown_connections 2024-03-29T04:08:59.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f112dbab640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 0x7f11200e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f508001a070 con 0x7f508815b900 2024-03-29T04:08:59.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f112dbab640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e7360 0x7f11200e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f7fb7fff640 1 -- 172.21.15.156:0/595546308 wait complete. 2024-03-29T04:08:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f112dbab640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f112003c0c0 con 0x7f1120007190 2024-03-29T04:08:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6ea558e640 1 -- 172.21.15.156:0/3077909098 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e9802caa0 con 0x7f6ea0072220 2024-03-29T04:08:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f0b5b7fe640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0b6006c580 0x7f0b6006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f112dbab640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f11200e7b90 con 0x7f11200e7360 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f112dbab640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f112003c240 con 0x7f11200e31c0 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f1127fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 0x7f11200e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f1127fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 0x7f11200e5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42666/0 (socket says 172.21.15.156:42666) 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f1127fff640 1 -- 172.21.15.156:0/2433341099 learned_addr learned my addr 172.21.15.156:0/2433341099 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f0b727fc640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0b6006c580 0x7f0b6006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0b68057590 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f1127fff640 1 -- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e7360 msgr2=0x7f11200e5fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T04:08:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f5084ff9640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f507406c560 0x7f507406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7fdce3959640 1 -- 172.21.15.156:0/3790842239 shutdown_connections 2024-03-29T04:08:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7fdce3959640 1 -- 172.21.15.156:0/3790842239 wait complete. 2024-03-29T04:08:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1885188247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 msgr2=0x7f6cbc0085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 --2- 172.21.15.156:0/1885188247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc0085d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc038840 tx=0x7f6cb8034910 comp rx=0 tx=0).stop 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1885188247 shutdown_connections 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 --2- 172.21.15.156:0/1885188247 >> v2:172.21.15.156:3300/0 conn(0x7f6cbc0e42d0 0x7f6cbc006ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 --2- 172.21.15.156:0/1885188247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc0085d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1885188247 >> 172.21.15.156:0/1885188247 conn(0x7f6cbc01ef40 msgr2=0x7f6cbc055c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1885188247 shutdown_connections 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.912+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1885188247 wait complete. 2024-03-29T04:08:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.913+0000 7f5087fff640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f507406c560 0x7f507406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.913+0000 7f0b727fc640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0b6006c580 0x7f0b6006ea40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b640097a0 tx=0x7f0b64038000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.913+0000 7f112cba9640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120007190 0x7f11200e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.913+0000 7f6ea558e640 1 -- 172.21.15.156:0/3077909098 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98034440 con 0x7f6ea0072220 2024-03-29T04:08:59.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.913+0000 7f5087fff640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f507406c560 0x7f507406ea20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5078034000 tx=0x7f507802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f508008ac40 con 0x7f508815b900 2024-03-29T04:08:59.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f7fb7fff640 1 Processor -- start 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f112d3aa640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e7360 0x7f11200e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f1127fff640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e7360 0x7f11200e5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f1127fff640 1 -- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120007190 msgr2=0x7f11200e5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f1127fff640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120007190 0x7f11200e5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f1127fff640 1 -- 172.21.15.156:0/2433341099 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11200dd170 con 0x7f11200e31c0 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f112cba9640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120007190 0x7f11200e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f112d3aa640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e7360 0x7f11200e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:08:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f7fb7fff640 1 -- start start 2024-03-29T04:08:59.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7fdce3959640 1 Processor -- start 2024-03-29T04:08:59.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f6cc99ec640 1 Processor -- start 2024-03-29T04:08:59.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f6cc99ec640 1 -- start start 2024-03-29T04:08:59.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7fdce3959640 1 -- start start 2024-03-29T04:08:59.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f6ea87f4640 1 -- 172.21.15.156:0/3077909098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0072220 msgr2=0x7f6ea016d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/3077909098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0072220 0x7f6ea016d630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009dc0 tx=0x7f6e9802bd70 comp rx=0 tx=0).stop 2024-03-29T04:08:59.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.914+0000 7f1127fff640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 0x7f11200e5a80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f11280511f0 tx=0x7f112806c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128094670 con 0x7f11200e31c0 2024-03-29T04:08:59.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112808f070 con 0x7f11200e31c0 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128094af0 con 0x7f11200e31c0 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11200dd450 con 0x7f11200e31c0 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11200dda10 con 0x7f11200e31c0 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f6ea87f4640 1 -- 172.21.15.156:0/3077909098 shutdown_connections 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fb00d0b60 0x7f7fb00ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb7fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb00ce640 0x7f7fb004d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb004d5e0 0x7f7fb004a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb7fff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb0037a80 con 0x7f7fb004d5e0 2024-03-29T04:08:59.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb7fff640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb0037900 con 0x7f7fb00d0b60 2024-03-29T04:08:59.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb7fff640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb0050190 con 0x7f7fb00ce640 2024-03-29T04:08:59.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/3077909098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0072220 0x7f6ea016d630 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f6ea87f4640 1 -- 172.21.15.156:0/3077909098 >> 172.21.15.156:0/3077909098 conn(0x7f6ea00674a0 msgr2=0x7f6ea006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6ea87f4640 1 -- 172.21.15.156:0/3077909098 shutdown_connections 2024-03-29T04:08:59.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb004d5e0 0x7f7fb004a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb004d5e0 0x7f7fb004a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45534/0 (socket says 172.21.15.156:45534) 2024-03-29T04:08:59.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb77fe640 1 -- 172.21.15.156:0/4050656973 learned_addr learned my addr 172.21.15.156:0/4050656973 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb00ce640 0x7f7fb004d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.915+0000 7f7fb6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fb00d0b60 0x7f7fb00ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce3959640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc06dba0 0x7fdcdc1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc99ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc049780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce3959640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc1639f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc99ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc0e42d0 0x7f6cbc006d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce3959640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0d0280 0x7fdcdc163f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce3959640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc162c50 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6ea87f4640 1 -- 172.21.15.156:0/3077909098 wait complete. 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc89ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc049780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc89ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc049780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35320/0 (socket says 172.21.15.156:35320) 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc89ea640 1 -- 172.21.15.156:0/1086431128 learned_addr learned my addr 172.21.15.156:0/1086431128 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc99ec640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6cbc0d2fe0 0x7f6cbc034d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce0ef4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc1639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce0ef4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc1639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45548/0 (socket says 172.21.15.156:45548) 2024-03-29T04:08:59.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce0ef4640 1 -- 172.21.15.156:0/2412311739 learned_addr learned my addr 172.21.15.156:0/2412311739 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc072210 con 0x7fdcdc0d0280 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb67fc640 1 -- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fb00d0b60 msgr2=0x7f7fb00ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fb00d0b60 0x7f7fb00ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb67fc640 1 -- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb004d5e0 msgr2=0x7f7fb004a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb004d5e0 0x7f7fb004a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f7fb67fc640 1 -- 172.21.15.156:0/4050656973 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb004de20 con 0x7f7fb00ce640 2024-03-29T04:08:59.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdcdc162dd0 con 0x7fdcdc06dba0 2024-03-29T04:08:59.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc008b10 con 0x7f6cbc0e42d0 2024-03-29T04:08:59.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.916+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc056b50 con 0x7f6cbc0dfef0 2024-03-29T04:08:59.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc0e4910 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc3fff640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc0e42d0 0x7f6cbc006d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6cbc0d2fe0 0x7f6cbc034d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7fdce0ef4640 1 -- 172.21.15.156:0/2412311739 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc06dba0 msgr2=0x7fdcdc1609f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T04:08:59.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 -- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 msgr2=0x7f6cbc049780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc049780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 -- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc0e42d0 msgr2=0x7f6cbc006d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc0e42d0 0x7f6cbc006d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7fdce0ef4640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc06dba0 0x7fdcdc1609f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc033f40 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7fdce0ef4640 1 -- 172.21.15.156:0/2412311739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0d0280 msgr2=0x7fdcdc163f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T04:08:59.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc3fff640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc0e42d0 0x7f6cbc006d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:59.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7fdce0ef4640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0d0280 0x7fdcdc163f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7fdce0ef4640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc1646b0 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7fdce0ef4640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc1639f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40090b0 tx=0x7fdcd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5080058c00 con 0x7f508815b900 2024-03-29T04:08:59.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1128094c90 con 0x7f11200e31c0 2024-03-29T04:08:59.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f7fb67fc640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb00ce640 0x7f7fb004d0a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7fac010810 tx=0x7f7fac010cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f6cc91eb640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6cbc0d2fe0 0x7f6cbc034d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6cc404fe10 tx=0x7f6cc406c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4099070 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc4069190 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f6ea87f4640 1 Processor -- start 2024-03-29T04:08:59.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f11257fa640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f110c06c600 0x7f110c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1128106a70 con 0x7f11200e31c0 2024-03-29T04:08:59.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f112cba9640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f110c06c600 0x7f110c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11200057e0 con 0x7f11200e31c0 2024-03-29T04:08:59.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.917+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac01a020 con 0x7f7fb00ce640 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb004e100 con 0x7f7fb00ce640 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb004e6c0 con 0x7f7fb00ce640 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fac00c930 con 0x7f7fb00ce640 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.918+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac01a020 con 0x7f7fb00ce640 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4073870 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc034220 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc0d2ab0 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f6ea87f4640 1 -- start start 2024-03-29T04:08:59.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f112cba9640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f110c06c600 0x7f110c06eac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f111c036000 tx=0x7f111c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.919+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f98005290 con 0x7f7fb00ce640 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea87f4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea0072220 0x7f6ea0145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea87f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 0x7f6ea015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea87f4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0162970 0x7f6ea01fc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea87f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea0162ed0 con 0x7f6ea00cda30 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea87f4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea0162d50 con 0x7f6ea0072220 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea87f4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea00d0360 con 0x7f6ea0162970 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea5d8f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 0x7f6ea015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea5d8f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 0x7f6ea015eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45558/0 (socket says 172.21.15.156:45558) 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea5d8f640 1 -- 172.21.15.156:0/793829267 learned_addr learned my addr 172.21.15.156:0/793829267 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea6d91640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0162970 0x7f6ea01fc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6ea6590640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea0072220 0x7f6ea0145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.920+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc0057e0 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6d91640 1 -- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea0072220 msgr2=0x7f6ea0145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6d91640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea0072220 0x7f6ea0145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6d91640 1 -- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 msgr2=0x7f6ea015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6d91640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 0x7f6ea015eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6d91640 1 -- 172.21.15.156:0/793829267 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea015dc40 con 0x7f6ea0162970 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea5d8f640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 0x7f6ea015eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6590640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea0072220 0x7f6ea0145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:59.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.921+0000 7f6ea6d91640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0162970 0x7f6ea01fc230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e98033000 tx=0x7f6e98004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.922+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4040070 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.922+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd4039bf0 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.922+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcdc164990 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.922+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4035040 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.922+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0b6801f020 con 0x7f0b6c0e12b0 2024-03-29T04:08:59.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.922+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcdc09a7d0 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.923+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcdc09e3b0 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.924+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdcd404b060 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.924+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98044020 con 0x7f6ea0162970 2024-03-29T04:08:59.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.924+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f11280d5070 con 0x7f11200e31c0 2024-03-29T04:08:59.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.925+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea015df20 con 0x7f6ea0162970 2024-03-29T04:08:59.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.925+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea014cab0 con 0x7f6ea0162970 2024-03-29T04:08:59.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.925+0000 7fdcd1ffb640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc006c530 0x7fdcc006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.926+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea015e440 con 0x7f6ea0162970 2024-03-29T04:08:59.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.925+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdcd40ad0e0 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.926+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7fac01d020 con 0x7f7fb00ce640 2024-03-29T04:08:59.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.927+0000 7f7f9ffff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fa406c960 0x7f7fa406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.927+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6cc409c070 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.927+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7fac00c730 con 0x7f7fb00ce640 2024-03-29T04:08:59.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.927+0000 7f7fb6ffd640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fa406c960 0x7f7fa406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.927+0000 7f6cc17fa640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6ca806c600 0x7f6ca806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.928+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6cc41064a0 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.928+0000 7f6cc89ea640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6ca806c600 0x7f6ca806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.928+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e9802ca50 con 0x7f6ea0162970 2024-03-29T04:08:59.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.928+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98039040 con 0x7f6ea0162970 2024-03-29T04:08:59.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.928+0000 7f7fb6ffd640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fa406c960 0x7f7fa406ee20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8034000 tx=0x7f7fa8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.928+0000 7f6cc89ea640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6ca806c600 0x7f6ca806eac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800fda0 tx=0x7f6cb803a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.926+0000 7fdce16f5640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc006c530 0x7fdcc006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.927+0000 7fdce16f5640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc006c530 0x7fdcc006e9f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009e70 tx=0x7fdcd8009830 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.930+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6cc40a3080 con 0x7f6cbc0d2fe0 2024-03-29T04:08:59.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.934+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdcd407b660 con 0x7fdcdc0b3d80 2024-03-29T04:08:59.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.934+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7fac024020 con 0x7f7fb00ce640 2024-03-29T04:08:59.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.940+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6e98047020 con 0x7f6ea0162970 2024-03-29T04:08:59.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.940+0000 7f6e96ffd640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e8406c560 0x7f6e8406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.941+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6e980ae2c0 con 0x7f6ea0162970 2024-03-29T04:08:59.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.941+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6ea015e440 con 0x7f6ea0162970 2024-03-29T04:08:59.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.941+0000 7f6ea6590640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e8406c560 0x7f6e8406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.942+0000 7f6ea6590640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e8406c560 0x7f6e8406ea20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e90000f60 tx=0x7f6e90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.996+0000 7fd73297c640 1 Processor -- start 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.996+0000 7fd73297c640 1 -- start start 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd73297c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd72c160500 0x7fd72c06f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd73297c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd72c0b4490 0x7fd72c06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd73297c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 0x7fd72c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd73297c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c163020 con 0x7fd72c160500 2024-03-29T04:08:59.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd73297c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c06fc80 con 0x7fd72c0b4490 2024-03-29T04:08:59.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd73297c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c0afc90 con 0x7fd72c06baf0 2024-03-29T04:08:59.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd72bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd72c160500 0x7fd72c06f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd730f19640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 0x7fd72c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd730f19640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 0x7fd72c06e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:42702/0 (socket says 172.21.15.156:42702) 2024-03-29T04:08:59.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.997+0000 7fd730f19640 1 -- 172.21.15.156:0/1977276518 learned_addr learned my addr 172.21.15.156:0/1977276518 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd72b7fe640 1 --2- 172.21.15.156:0/1977276518 >> v2:172.21.15.203:3300/0 conn(0x7fd72c0b4490 0x7fd72c06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd730f19640 1 -- 172.21.15.156:0/1977276518 >> v2:172.21.15.203:3300/0 conn(0x7fd72c0b4490 msgr2=0x7fd72c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd730f19640 1 --2- 172.21.15.156:0/1977276518 >> v2:172.21.15.203:3300/0 conn(0x7fd72c0b4490 0x7fd72c06b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd730f19640 1 -- 172.21.15.156:0/1977276518 >> v2:172.21.15.156:3300/0 conn(0x7fd72c160500 msgr2=0x7fd72c06f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:08:59.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd730f19640 1 --2- 172.21.15.156:0/1977276518 >> v2:172.21.15.156:3300/0 conn(0x7fd72c160500 0x7fd72c06f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd730f19640 1 -- 172.21.15.156:0/1977276518 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c0aff70 con 0x7fd72c06baf0 2024-03-29T04:08:59.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.998+0000 7fd72bfff640 1 --2- 172.21.15.156:0/1977276518 >> v2:172.21.15.156:3300/0 conn(0x7fd72c160500 0x7fd72c06f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:08:59.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.999+0000 7fd730f19640 1 --2- 172.21.15.156:0/1977276518 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 0x7fd72c06e390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd72c06d230 tx=0x7fd72402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85750451f50b8a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:08:59.999+0000 7fd72affd640 1 -- 172.21.15.156:0/1977276518 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72402ca80 con 0x7fd72c06baf0 2024-03-29T04:08:59.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.000+0000 7fd72affd640 1 -- 172.21.15.156:0/1977276518 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd72402cc20 con 0x7fd72c06baf0 2024-03-29T04:08:59.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.000+0000 7fd72affd640 1 -- 172.21.15.156:0/1977276518 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7240343f0 con 0x7fd72c06baf0 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.000+0000 7fd73297c640 1 -- 172.21.15.156:0/1977276518 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 msgr2=0x7fd72c06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.000+0000 7fd73297c640 1 --2- 172.21.15.156:0/1977276518 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 0x7fd72c06e390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd72c06d230 tx=0x7fd72402bee0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 -- 172.21.15.156:0/1977276518 shutdown_connections 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 --2- 172.21.15.156:0/1977276518 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c06baf0 0x7fd72c06e390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 --2- 172.21.15.156:0/1977276518 >> v2:172.21.15.203:3300/0 conn(0x7fd72c0b4490 0x7fd72c06b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 --2- 172.21.15.156:0/1977276518 >> v2:172.21.15.156:3300/0 conn(0x7fd72c160500 0x7fd72c06f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 -- 172.21.15.156:0/1977276518 >> 172.21.15.156:0/1977276518 conn(0x7fd72c0b2de0 msgr2=0x7fd72c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:08:59.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 -- 172.21.15.156:0/1977276518 shutdown_connections 2024-03-29T04:08:59.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.001+0000 7fd73297c640 1 -- 172.21.15.156:0/1977276518 wait complete. 2024-03-29T04:08:59.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.003+0000 7fd73297c640 1 Processor -- start 2024-03-29T04:08:59.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.003+0000 7fd73297c640 1 -- start start 2024-03-29T04:08:59.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.004+0000 7fd73297c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c06baf0 0x7fd72c0af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.004+0000 7fd73297c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c0b4490 0x7fd72c164ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.004+0000 7fd73297c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 0x7fd72c165000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:08:59.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.004+0000 7fd73297c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0b4920 con 0x7fd72c06baf0 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.004+0000 7fd73297c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0b0250 con 0x7fd72c160500 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.004+0000 7fd73297c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c06e8d0 con 0x7fd72c0b4490 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 0x7fd72c165000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c0b4490 0x7fd72c164ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 0x7fd72c165000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:35374/0 (socket says 172.21.15.156:35374) 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 -- 172.21.15.156:0/253367821 learned_addr learned my addr 172.21.15.156:0/253367821 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd72bfff640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c06baf0 0x7fd72c0af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 -- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c0b4490 msgr2=0x7fd72c164ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c0b4490 0x7fd72c164ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 -- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c06baf0 msgr2=0x7fd72c0af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c06baf0 0x7fd72c0af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 -- 172.21.15.156:0/253367821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c158530 con 0x7fd72c160500 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd72b7fe640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c0b4490 0x7fd72c164ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:59.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd72bfff640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c06baf0 0x7fd72c0af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:08:59.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.005+0000 7fd730f19640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 0x7fd72c165000 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd724036040 tx=0x7fd724004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:08:59.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.006+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724065020 con 0x7fd72c160500 2024-03-29T04:08:59.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.006+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd724007e50 con 0x7fd72c160500 2024-03-29T04:08:59.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.007+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724060680 con 0x7fd72c160500 2024-03-29T04:08:59.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.007+0000 7fd73297c640 1 -- 172.21.15.156:0/253367821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c158810 con 0x7fd72c160500 2024-03-29T04:08:59.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.008+0000 7fd73297c640 1 -- 172.21.15.156:0/253367821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c158dd0 con 0x7fd72c160500 2024-03-29T04:09:00.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.024+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd704005290 con 0x7fd72c160500 2024-03-29T04:09:00.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.027+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd724068020 con 0x7fd72c160500 2024-03-29T04:09:00.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.031+0000 7fd728ff9640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd71806c600 0x7fd71806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:00.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.032+0000 7fd72bfff640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd71806c600 0x7fd71806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:00.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.033+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd7240cff80 con 0x7fd72c160500 2024-03-29T04:09:00.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.037+0000 7fd72bfff640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd71806c600 0x7fd71806eac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd72c165700 tx=0x7fd71c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:00.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.037+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd72406f070 con 0x7fd72c160500 2024-03-29T04:09:00.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.304+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fdcdc04f9d0 con 0x7fdcdc0b3d80 2024-03-29T04:09:00.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.318+0000 7fdcd1ffb640 1 -- 172.21.15.156:0/2412311739 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdcd407af60 con 0x7fdcdc0b3d80 2024-03-29T04:09:00.312 INFO:teuthology.orchestra.run.smithi156.stdout:55834574853 2024-03-29T04:09:00.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.336+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc006c530 msgr2=0x7fdcc006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.336+0000 7fdce3959640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc006c530 0x7fdcc006e9f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009e70 tx=0x7fdcd8009830 comp rx=0 tx=0).stop 2024-03-29T04:09:00.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.336+0000 7f6cc99ec640 1 -- 172.21.15.156:0/1086431128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6cbc00f3c0 con 0x7f6cbc0d2fe0 2024-03-29T04:09:00.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.336+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 msgr2=0x7fdcdc1639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.336+0000 7fdce3959640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc1639f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40090b0 tx=0x7fdcd40046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.338+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 shutdown_connections 2024-03-29T04:09:00.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.338+0000 7fdce3959640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc006c530 0x7fdcc006e9f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.338+0000 7fdce3959640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcdc0d0280 0x7fdcdc163f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.338+0000 7fdce3959640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcdc0b3d80 0x7fdcdc1639f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.338+0000 7fdce3959640 1 --2- 172.21.15.156:0/2412311739 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdcdc06dba0 0x7fdcdc1609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.338+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 >> 172.21.15.156:0/2412311739 conn(0x7fdcdc0b1f30 msgr2=0x7fdcdc15f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.321 INFO:teuthology.orchestra.run.smithi156.stdout:55834574853 2024-03-29T04:09:00.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.336+0000 7f6cc17fa640 1 -- 172.21.15.156:0/1086431128 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6cc4091070 con 0x7f6cbc0d2fe0 2024-03-29T04:09:00.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.339+0000 7f6caeffd640 1 -- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6ca806c600 msgr2=0x7f6ca806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.339+0000 7f6caeffd640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6ca806c600 0x7f6ca806eac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800fda0 tx=0x7f6cb803a000 comp rx=0 tx=0).stop 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.339+0000 7f6caeffd640 1 -- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6cbc0d2fe0 msgr2=0x7f6cbc034d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.339+0000 7f6caeffd640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6cbc0d2fe0 0x7f6cbc034d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6cc404fe10 tx=0x7f6cc406c990 comp rx=0 tx=0).stop 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.340+0000 7f6caeffd640 1 -- 172.21.15.156:0/1086431128 shutdown_connections 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.340+0000 7f6caeffd640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6ca806c600 0x7f6ca806eac0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800fda0 tx=0x7f6cb803a000 comp rx=0 tx=0).stop 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.340+0000 7f6caeffd640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6cbc0d2fe0 0x7f6cbc034d00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.340+0000 7f6caeffd640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc0e42d0 0x7f6cbc006d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.340+0000 7f6caeffd640 1 --2- 172.21.15.156:0/1086431128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cbc0dfef0 0x7f6cbc049780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.340+0000 7f6caeffd640 1 -- 172.21.15.156:0/1086431128 >> 172.21.15.156:0/1086431128 conn(0x7f6cbc01ef40 msgr2=0x7f6cbc0549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.341+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 shutdown_connections 2024-03-29T04:09:00.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.341+0000 7f6caeffd640 1 -- 172.21.15.156:0/1086431128 shutdown_connections 2024-03-29T04:09:00.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.341+0000 7f6caeffd640 1 -- 172.21.15.156:0/1086431128 wait complete. 2024-03-29T04:09:00.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.343+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1120056ca0 con 0x7f11200e31c0 2024-03-29T04:09:00.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.343+0000 7fdce3959640 1 -- 172.21.15.156:0/2412311739 wait complete. 2024-03-29T04:09:00.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.348+0000 7f11257fa640 1 -- 172.21.15.156:0/2433341099 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f11280d4970 con 0x7f11200e31c0 2024-03-29T04:09:00.328 INFO:teuthology.orchestra.run.smithi156.stdout:60129542149 2024-03-29T04:09:00.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.354+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f110c06c600 msgr2=0x7f110c06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.354+0000 7f112dbab640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f110c06c600 0x7f110c06eac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f111c036000 tx=0x7f111c008030 comp rx=0 tx=0).stop 2024-03-29T04:09:00.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.354+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 msgr2=0x7f11200e5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.354+0000 7f112dbab640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 0x7f11200e5a80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f11280511f0 tx=0x7f112806c990 comp rx=0 tx=0).stop 2024-03-29T04:09:00.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.354+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7f98005850 con 0x7f7fb00ce640 2024-03-29T04:09:00.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 shutdown_connections 2024-03-29T04:09:00.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f112dbab640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f110c06c600 0x7f110c06eac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f112dbab640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11200e7360 0x7f11200e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f112dbab640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11200e31c0 0x7f11200e5a80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f112dbab640 1 --2- 172.21.15.156:0/2433341099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120007190 0x7f11200e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 >> 172.21.15.156:0/2433341099 conn(0x7f112003aa30 msgr2=0x7f112003da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.355+0000 7f7f9ffff640 1 -- 172.21.15.156:0/4050656973 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7fac05a390 con 0x7f7fb00ce640 2024-03-29T04:09:00.336 INFO:teuthology.orchestra.run.smithi156.stdout:60129542150 2024-03-29T04:09:00.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fa406c960 msgr2=0x7f7fa406ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fa406c960 0x7f7fa406ee20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8034000 tx=0x7f7fa8009030 comp rx=0 tx=0).stop 2024-03-29T04:09:00.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb00ce640 msgr2=0x7f7fb004d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb00ce640 0x7f7fb004d0a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7fac010810 tx=0x7f7fac010cd0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 shutdown_connections 2024-03-29T04:09:00.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fa406c960 0x7f7fa406ee20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb004d5e0 0x7f7fb004a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fb00ce640 0x7f7fb004d0a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 --2- 172.21.15.156:0/4050656973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fb00d0b60 0x7f7fb00ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 >> 172.21.15.156:0/4050656973 conn(0x7f7fb0009fa0 msgr2=0x7f7fb00d3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 shutdown_connections 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.359+0000 7f7fb7fff640 1 -- 172.21.15.156:0/4050656973 wait complete. 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.360+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 shutdown_connections 2024-03-29T04:09:00.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.360+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0b6c005eb0 con 0x7f0b6c0e12b0 2024-03-29T04:09:00.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.360+0000 7f112dbab640 1 -- 172.21.15.156:0/2433341099 wait complete. 2024-03-29T04:09:00.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.361+0000 7f0b5b7fe640 1 -- 172.21.15.156:0/2432854198 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0b68011e20 con 0x7f0b6c0e12b0 2024-03-29T04:09:00.341 INFO:teuthology.orchestra.run.smithi156.stdout:55834574854 2024-03-29T04:09:00.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0b6006c580 msgr2=0x7f0b6006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0b6006c580 0x7f0b6006ea40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b640097a0 tx=0x7f0b64038000 comp rx=0 tx=0).stop 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b6c0e12b0 msgr2=0x7f0b6c04a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b6c0e12b0 0x7f0b6c04a8a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009fd0 tx=0x7f0b6800b870 comp rx=0 tx=0).stop 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 shutdown_connections 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0b6006c580 0x7f0b6006ea40 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b640097a0 tx=0x7f0b64038000 comp rx=0 tx=0).stop 2024-03-29T04:09:00.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b6c0d7530 0x7f0b6c0dbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b6c0e4d30 0x7f0b6c038100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 --2- 172.21.15.156:0/2432854198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b6c0e12b0 0x7f0b6c04a8a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.364+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 >> 172.21.15.156:0/2432854198 conn(0x7f0b6c020040 msgr2=0x7f0b6c0dd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-03-29T04:09:00.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.365+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 shutdown_connections 2024-03-29T04:09:00.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.365+0000 7f0b73fff640 1 -- 172.21.15.156:0/2432854198 wait complete. 2024-03-29T04:09:00.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.366+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5068005580 con 0x7f508815b900 2024-03-29T04:09:00.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.367+0000 7f5084ff9640 1 -- 172.21.15.156:0/17132800 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f508005cf00 con 0x7f508815b900 2024-03-29T04:09:00.347 INFO:teuthology.orchestra.run.smithi156.stdout:60129542151 2024-03-29T04:09:00.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.372+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f507406c560 msgr2=0x7f507406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.372+0000 7f508d6f7640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f507406c560 0x7f507406ea20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5078034000 tx=0x7f507802f000 comp rx=0 tx=0).stop 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 msgr2=0x7f50880b0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 0x7f50880b0cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f508000c780 tx=0x7f508000cc40 comp rx=0 tx=0).stop 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 shutdown_connections 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f507406c560 0x7f507406ea20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508815b900 0x7f50880b0cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5088155820 0x7f5088152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 --2- 172.21.15.156:0/17132800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f508806ee00 0x7f508814fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 >> 172.21.15.156:0/17132800 conn(0x7f508806c6c0 msgr2=0x7f508815ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 shutdown_connections 2024-03-29T04:09:00.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.373+0000 7f508d6f7640 1 -- 172.21.15.156:0/17132800 wait complete. 2024-03-29T04:09:00.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.7 2024-03-29T04:09:00.361 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.4 2024-03-29T04:09:00.361 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-29T04:09:00.362 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:00.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-03-29T04:09:00.369 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.389+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6ea0073450 con 0x7f6ea0162970 2024-03-29T04:09:00.369 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.390+0000 7f6e96ffd640 1 -- 172.21.15.156:0/793829267 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6e9804e030 con 0x7f6ea0162970 2024-03-29T04:09:00.370 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:00.370 INFO:teuthology.orchestra.run.smithi156.stdout:55834574853 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e8406c560 msgr2=0x7f6e8406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e8406c560 0x7f6e8406ea20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e90000f60 tx=0x7f6e90009040 comp rx=0 tx=0).stop 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0162970 msgr2=0x7f6ea01fc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0162970 0x7f6ea01fc230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e98033000 tx=0x7f6e98004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 shutdown_connections 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e8406c560 0x7f6e8406ea20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ea0162970 0x7f6ea01fc230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ea00cda30 0x7f6ea015eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 --2- 172.21.15.156:0/793829267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea0072220 0x7f6ea0145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 >> 172.21.15.156:0/793829267 conn(0x7f6ea00674a0 msgr2=0x7f6ea00b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 shutdown_connections 2024-03-29T04:09:00.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.392+0000 7f6ea87f4640 1 -- 172.21.15.156:0/793829267 wait complete. 2024-03-29T04:09:00.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-29T04:09:00.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.506+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd704005580 con 0x7fd72c160500 2024-03-29T04:09:00.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.507+0000 7fd728ff9640 1 -- 172.21.15.156:0/253367821 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd72409e650 con 0x7fd72c160500 2024-03-29T04:09:00.486 INFO:teuthology.orchestra.run.smithi156.stdout:60129542149 2024-03-29T04:09:00.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.508+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd71806c600 msgr2=0x7fd71806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.508+0000 7fd7127fc640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd71806c600 0x7fd71806eac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd72c165700 tx=0x7fd71c008040 comp rx=0 tx=0).stop 2024-03-29T04:09:00.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.508+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 msgr2=0x7fd72c165000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.508+0000 7fd7127fc640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 0x7fd72c165000 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd724036040 tx=0x7fd724004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 shutdown_connections 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd71806c600 0x7fd71806eac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c160500 0x7fd72c165000 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd72c0b4490 0x7fd72c164ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 --2- 172.21.15.156:0/253367821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c06baf0 0x7fd72c0af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 >> 172.21.15.156:0/253367821 conn(0x7fd72c0b2de0 msgr2=0x7fd72c15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 shutdown_connections 2024-03-29T04:09:00.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:00.509+0000 7fd7127fc640 1 -- 172.21.15.156:0/253367821 wait complete. 2024-03-29T04:09:00.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.6 2024-03-29T04:09:01.343 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T04:09:01.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T04:09:01.361 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-29T04:09:01.363 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-29T04:09:01.383 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-29T04:09:01.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.502+0000 7f413964a640 1 Processor -- start 2024-03-29T04:09:01.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.502+0000 7f413964a640 1 -- start start 2024-03-29T04:09:01.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.503+0000 7f413964a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f41340cd940 0x7f413409aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.503+0000 7f413964a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f41340cddc0 0x7f413409af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f413964a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41340d01e0 con 0x7f413415cb50 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f413964a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f41340d04c0 con 0x7f41340cd940 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f413964a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f41340d07a0 con 0x7f41340cddc0 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f41340cd940 0x7f413409aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f41340cd940 0x7f413409aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45578/0 (socket says 172.21.15.156:45578) 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 -- 172.21.15.156:0/2851065774 learned_addr learned my addr 172.21.15.156:0/2851065774 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f41337fe640 1 --2- 172.21.15.156:0/2851065774 >> v2:172.21.15.203:3300/0 conn(0x7f41340cddc0 0x7f413409af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 -- 172.21.15.156:0/2851065774 >> v2:172.21.15.203:3300/0 conn(0x7f41340cddc0 msgr2=0x7f413409af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:01.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 --2- 172.21.15.156:0/2851065774 >> v2:172.21.15.203:3300/0 conn(0x7f41340cddc0 0x7f413409af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f41327fc640 1 -- 172.21.15.156:0/2851065774 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346800744 0 0) 0x7f41340d01e0 con 0x7f413415cb50 2024-03-29T04:09:01.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 -- 172.21.15.156:0/2851065774 >> v1:172.21.15.203:6789/0 conn(0x7f413415cb50 legacy=0x7f413415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.504+0000 7f412bfff640 1 -- 172.21.15.156:0/2851065774 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41340d0a80 con 0x7f41340cd940 2024-03-29T04:09:01.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.506+0000 7f412bfff640 1 --2- 172.21.15.156:0/2851065774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cd940 0x7f413409aa40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f41340ce750 tx=0x7f411c02bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40d87ef787ca634 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.506+0000 7f41327fc640 1 -- 172.21.15.156:0/2851065774 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c039070 con 0x7f41340cd940 2024-03-29T04:09:01.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.506+0000 7f41327fc640 1 -- 172.21.15.156:0/2851065774 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c02c730 con 0x7f41340cd940 2024-03-29T04:09:01.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.506+0000 7f41327fc640 1 -- 172.21.15.156:0/2851065774 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c02cb00 con 0x7f41340cd940 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 -- 172.21.15.156:0/2851065774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cd940 msgr2=0x7f413409aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 --2- 172.21.15.156:0/2851065774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cd940 0x7f413409aa40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f41340ce750 tx=0x7f411c02bb90 comp rx=0 tx=0).stop 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 -- 172.21.15.156:0/2851065774 shutdown_connections 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 --2- 172.21.15.156:0/2851065774 >> v2:172.21.15.203:3300/0 conn(0x7f41340cddc0 0x7f413409af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 --2- 172.21.15.156:0/2851065774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cd940 0x7f413409aa40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 -- 172.21.15.156:0/2851065774 >> 172.21.15.156:0/2851065774 conn(0x7f41340b3d60 msgr2=0x7f41340b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 -- 172.21.15.156:0/2851065774 shutdown_connections 2024-03-29T04:09:01.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.507+0000 7f413964a640 1 -- 172.21.15.156:0/2851065774 wait complete. 2024-03-29T04:09:01.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.510+0000 7f413964a640 1 Processor -- start 2024-03-29T04:09:01.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.510+0000 7f413964a640 1 -- start start 2024-03-29T04:09:01.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.511+0000 7f413964a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 0x7f413415c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.511+0000 7f413964a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f413415cb50 0x7f413415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.511+0000 7f413964a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41341584d0 0x7f41340b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f413964a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41340cffc0 con 0x7f41340cddc0 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f4132ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 0x7f413415c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f4132ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 0x7f413415c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45590/0 (socket says 172.21.15.156:45590) 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f412bfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f413415cb50 0x7f413415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f4132ffd640 1 -- 172.21.15.156:0/1016327848 learned_addr learned my addr 172.21.15.156:0/1016327848 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f41337fe640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41341584d0 0x7f41340b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f413964a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41340cfe40 con 0x7f41341584d0 2024-03-29T04:09:01.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.512+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f413409b4c0 con 0x7f413415cb50 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412bfff640 1 -- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41341584d0 msgr2=0x7f41340b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412bfff640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41341584d0 0x7f41340b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412bfff640 1 -- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 msgr2=0x7f413415c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412bfff640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 0x7f413415c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412bfff640 1 -- 172.21.15.156:0/1016327848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413415b810 con 0x7f413415cb50 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f4132ffd640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 0x7f413415c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412bfff640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f413415cb50 0x7f413415afe0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41340734f0 tx=0x7f411c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c043020 con 0x7f413415cb50 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c02cd10 con 0x7f413415cb50 2024-03-29T04:09:01.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.513+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c039040 con 0x7f413415cb50 2024-03-29T04:09:01.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.514+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41340b51d0 con 0x7f413415cb50 2024-03-29T04:09:01.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.514+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41340b5790 con 0x7f413415cb50 2024-03-29T04:09:01.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.515+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41340a03c0 con 0x7f413415cb50 2024-03-29T04:09:01.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.515+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f411c046020 con 0x7f413415cb50 2024-03-29T04:09:01.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.516+0000 7f412b7fe640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f411406c530 0x7f411406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.516+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f411c0ad220 con 0x7f413415cb50 2024-03-29T04:09:01.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.516+0000 7f4132ffd640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f411406c530 0x7f411406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.517+0000 7f4132ffd640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f411406c530 0x7f411406e9f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4124007fd0 tx=0x7f4124009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.499 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-29T04:09:01.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.521+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f411c04d030 con 0x7f413415cb50 2024-03-29T04:09:01.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.528+0000 7f85af559640 1 Processor -- start 2024-03-29T04:09:01.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.528+0000 7f85af559640 1 -- start start 2024-03-29T04:09:01.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.528+0000 7f2923ea7640 1 Processor -- start 2024-03-29T04:09:01.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.528+0000 7f2923ea7640 1 -- start start 2024-03-29T04:09:01.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.529+0000 7f85af559640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f85a806b5b0 0x7f85a80b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f85af559640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a80b2b10 con 0x7f85a80b40d0 2024-03-29T04:09:01.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f85af559640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a80b5f20 con 0x7f85a8160460 2024-03-29T04:09:01.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f85af559640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f85a815efc0 con 0x7f85a806b5b0 2024-03-29T04:09:01.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f85acaf4640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f85a80b40d0 0x7f85a806b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47492/0 (socket says 172.21.15.156:47492) 2024-03-29T04:09:01.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2923ea7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f291c0d0280 0x7f291c0d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2923ea7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c072730 0x7f291c072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f85acaf4640 1 -- 172.21.15.156:0/737729482 learned_addr learned my addr 172.21.15.156:0/737729482 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2923ea7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c0d0ba0 con 0x7f291c0b3d80 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2923ea7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f291c06dd70 con 0x7f291c0d0280 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2923ea7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f291c06e300 con 0x7f291c072730 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2921c43640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f291c0d0280 0x7f291c0d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2921c43640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f291c0d0280 0x7f291c0d0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:38226/0 (socket says 172.21.15.156:38226) 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2921c43640 1 -- 172.21.15.156:0/1670606317 learned_addr learned my addr 172.21.15.156:0/1670606317 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f2922444640 1 --2- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c072730 0x7f291c072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.530+0000 7f85ad2f5640 1 --2- 172.21.15.156:0/737729482 >> v2:172.21.15.203:3300/0 conn(0x7f85a806b5b0 0x7f85a80b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632379446 0 0) 0x7f85a80b5f20 con 0x7f85a8160460 2024-03-29T04:09:01.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f2921c43640 1 -- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c072730 msgr2=0x7f291c072ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:01.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f2921c43640 1 --2- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c072730 0x7f291c072ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f2921c43640 1 -- 172.21.15.156:0/1670606317 >> v1:172.21.15.203:6789/0 conn(0x7f291c0b3d80 legacy=0x7f291c06d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f2921c43640 1 -- 172.21.15.156:0/1670606317 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c06e020 con 0x7f291c0d0280 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8594003520 con 0x7f85a8160460 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791577848 0 0) 0x7f85a80b2b10 con 0x7f85a80b40d0 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85a80b5f20 con 0x7f85a80b40d0 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2071016125 0 0) 0x7f85a80b5f20 con 0x7f85a80b40d0 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 >> v2:172.21.15.203:3300/0 conn(0x7f85a806b5b0 msgr2=0x7f85a80b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 --2- 172.21.15.156:0/737729482 >> v2:172.21.15.203:3300/0 conn(0x7f85a806b5b0 0x7f85a80b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 >> v1:172.21.15.203:6789/0 conn(0x7f85a8160460 legacy=0x7f85a8168290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a80b6200 con 0x7f85a80b40d0 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f8598001670 con 0x7f85a80b40d0 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 >> v1:172.21.15.156:6789/0 conn(0x7f85a80b40d0 legacy=0x7f85a806b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594003b40 0x7f8594003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8594004800 0x7f8594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85940051c0 0x7f859400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85a815efc0 con 0x7f8594004800 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8594014850 con 0x7f8594003b40 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.531+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8594014a50 con 0x7f85940051c0 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85adaf6640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85940051c0 0x7f859400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85ad2f5640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594003b40 0x7f8594003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85adaf6640 1 -- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594003b40 msgr2=0x7f8594003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85adaf6640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594003b40 0x7f8594003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85adaf6640 1 -- 172.21.15.156:0/737729482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8594004800 msgr2=0x7f8594004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T04:09:01.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85acaf4640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8594004800 0x7f8594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f2921c43640 1 --2- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c0d0280 0x7f291c0d0660 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f290c009870 tx=0x7f290c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614e7614c71c4b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85adaf6640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8594004800 0x7f8594004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85adaf6640 1 -- 172.21.15.156:0/737729482 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a80b6200 con 0x7f85940051c0 2024-03-29T04:09:01.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f85acaf4640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8594004800 0x7f8594004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:01.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.532+0000 7f2920c41640 1 -- 172.21.15.156:0/1670606317 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c035020 con 0x7f291c0d0280 2024-03-29T04:09:01.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.533+0000 7f2920c41640 1 -- 172.21.15.156:0/1670606317 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f290c031640 con 0x7f291c0d0280 2024-03-29T04:09:01.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.533+0000 7f2920c41640 1 -- 172.21.15.156:0/1670606317 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c035020 con 0x7f291c0d0280 2024-03-29T04:09:01.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f85adaf6640 1 -- 172.21.15.156:0/737729482 reap_dead start 2024-03-29T04:09:01.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 -- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c0d0280 msgr2=0x7f291c0d0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 --2- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c0d0280 0x7f291c0d0660 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f290c009870 tx=0x7f290c02b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 -- 172.21.15.156:0/1670606317 shutdown_connections 2024-03-29T04:09:01.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f85adaf6640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85940051c0 0x7f859400b9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85a0001fd0 tx=0x7f8594005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e1eea5ec6df1f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 --2- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c072730 0x7f291c072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 --2- 172.21.15.156:0/1670606317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c0d0280 0x7f291c0d0660 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0001c20 con 0x7f85940051c0 2024-03-29T04:09:01.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 -- 172.21.15.156:0/1670606317 >> 172.21.15.156:0/1670606317 conn(0x7f291c0b1f30 msgr2=0x7f291c0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0003b70 con 0x7f85940051c0 2024-03-29T04:09:01.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f859ffff640 1 -- 172.21.15.156:0/737729482 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a00050d0 con 0x7f85940051c0 2024-03-29T04:09:01.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 -- 172.21.15.156:0/1670606317 shutdown_connections 2024-03-29T04:09:01.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.534+0000 7f2923ea7640 1 -- 172.21.15.156:0/1670606317 wait complete. 2024-03-29T04:09:01.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.535+0000 7f85af559640 1 -- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85940051c0 msgr2=0x7f859400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.535+0000 7f85af559640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85940051c0 0x7f859400b9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85a0001fd0 tx=0x7f8594005830 comp rx=0 tx=0).stop 2024-03-29T04:09:01.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.535+0000 7f85af559640 1 -- 172.21.15.156:0/737729482 shutdown_connections 2024-03-29T04:09:01.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.535+0000 7f85af559640 1 --2- 172.21.15.156:0/737729482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85940051c0 0x7f859400b9c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.535+0000 7f85af559640 1 -- 172.21.15.156:0/737729482 >> 172.21.15.156:0/737729482 conn(0x7f85a80b65f0 msgr2=0x7f85a80b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.536+0000 7f85af559640 1 -- 172.21.15.156:0/737729482 shutdown_connections 2024-03-29T04:09:01.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.536+0000 7f2923ea7640 1 Processor -- start 2024-03-29T04:09:01.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.536+0000 7f85af559640 1 -- 172.21.15.156:0/737729482 wait complete. 2024-03-29T04:09:01.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.536+0000 7f2923ea7640 1 -- start start 2024-03-29T04:09:01.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2923ea7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c072730 0x7f291c161b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2923ea7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 0x7f291c06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2921442640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 0x7f291c06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2921442640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 0x7f291c06b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48442/0 (socket says 172.21.15.156:48442) 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2921442640 1 -- 172.21.15.156:0/2375512648 learned_addr learned my addr 172.21.15.156:0/2375512648 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c0d0280 0x7f291c06baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f291c072210 con 0x7f291c0b3d80 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f291c071f30 con 0x7f291c072730 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.537+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f291c162c50 con 0x7f291c0d0280 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921c43640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c072730 0x7f291c161b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f85af559640 1 Processor -- start 2024-03-29T04:09:01.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f85af559640 1 -- start start 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2922444640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c0d0280 0x7f291c06baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921442640 1 -- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c0d0280 msgr2=0x7f291c06baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921442640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c0d0280 0x7f291c06baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921442640 1 -- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c072730 msgr2=0x7f291c161b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921442640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c072730 0x7f291c161b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921442640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c15a470 con 0x7f291c0b3d80 2024-03-29T04:09:01.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.538+0000 7f2921c43640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c072730 0x7f291c161b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f2921442640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 0x7f291c06b5b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f291800c7b0 tx=0x7f291800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85af559640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85940051c0 0x7f85a81396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85af559640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85a81f4240 0x7f85a815e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85af559640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85a809c020 0x7f85a81f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85af559640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85a80b5ce0 con 0x7f85a809c020 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85af559640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85a80b5b60 con 0x7f85940051c0 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85af559640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f85a8147520 con 0x7f85a81f4240 2024-03-29T04:09:01.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180050d0 con 0x7f291c0b3d80 2024-03-29T04:09:01.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2918005270 con 0x7f291c0b3d80 2024-03-29T04:09:01.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291800a640 con 0x7f291c0b3d80 2024-03-29T04:09:01.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85acaf4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85a81f4240 0x7f85a815e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85acaf4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85a81f4240 0x7f85a815e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:48154/0 (socket says 172.21.15.156:48154) 2024-03-29T04:09:01.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.539+0000 7f85acaf4640 1 -- 172.21.15.156:0/195726090 learned_addr learned my addr 172.21.15.156:0/195726090 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85a809c020 0x7f85a81f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 -- 172.21.15.156:0/195726090 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85a81f4240 msgr2=0x7f85a815e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85a81f4240 0x7f85a815e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 -- 172.21.15.156:0/195726090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85940051c0 msgr2=0x7f85a81396d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85940051c0 0x7f85a81396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 -- 172.21.15.156:0/195726090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a809ca20 con 0x7f85a809c020 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291c15a750 con 0x7f291c0b3d80 2024-03-29T04:09:01.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f291c15ad10 con 0x7f291c0b3d80 2024-03-29T04:09:01.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.540+0000 7f85adaf6640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85a809c020 0x7f85a81f35c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f85a80b47b0 tx=0x7f85a0001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.541+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0001c20 con 0x7f85a809c020 2024-03-29T04:09:01.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.541+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0003970 con 0x7f85a809c020 2024-03-29T04:09:01.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.541+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0007db0 con 0x7f85a809c020 2024-03-29T04:09:01.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.541+0000 7f85af559640 1 -- 172.21.15.156:0/195726090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a809cd00 con 0x7f85a809c020 2024-03-29T04:09:01.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.542+0000 7f85af559640 1 -- 172.21.15.156:0/195726090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a809d2c0 con 0x7f85a809c020 2024-03-29T04:09:01.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.543+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f291c09e3b0 con 0x7f291c0b3d80 2024-03-29T04:09:01.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.543+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8584005290 con 0x7f85a809c020 2024-03-29T04:09:01.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.544+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f291800a800 con 0x7f291c0b3d80 2024-03-29T04:09:01.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.544+0000 7f29127fc640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f290006c630 0x7f290006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.544+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f291808b210 con 0x7f291c0b3d80 2024-03-29T04:09:01.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.545+0000 7f2921c43640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f290006c630 0x7f290006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.545+0000 7f2921c43640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f290006c630 0x7f290006eaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f290c036000 tx=0x7f290c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.545+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f85a002dad0 con 0x7f85a809c020 2024-03-29T04:09:01.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.546+0000 7f859dffb640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f858c06c630 0x7f858c06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.546+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f85a00aca40 con 0x7f85a809c020 2024-03-29T04:09:01.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.546+0000 7f85ad2f5640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f858c06c630 0x7f858c06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.546+0000 7f85ad2f5640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f858c06c630 0x7f858c06eaf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85a8147250 tx=0x7f85a4009320 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.553+0000 7f9bc1360640 1 Processor -- start 2024-03-29T04:09:01.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.553+0000 7f9bc1360640 1 -- start start 2024-03-29T04:09:01.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.554+0000 7f9bc1360640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 0x7f9bbc0b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.553+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f85a007a9f0 con 0x7f85a809c020 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.554+0000 7f9bc1360640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9bbc0b4310 0x7f9bbc0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bc1360640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bbc15cb10 con 0x7f9bbc0cf340 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 0x7f9bbc0b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 0x7f9bbc0b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:48168/0 (socket says 172.21.15.156:48168) 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bba7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9bbc0b4310 0x7f9bbc0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 -- 172.21.15.156:0/450015652 learned_addr learned my addr 172.21.15.156:0/450015652 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9bbc0b4770 con 0x7f9bbc0b4310 2024-03-29T04:09:01.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9bbc15cdf0 con 0x7f9bbc145c70 2024-03-29T04:09:01.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bb9ffb640 1 -- 172.21.15.156:0/450015652 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 111945972 0 0) 0x7f9bbc15cb10 con 0x7f9bbc0cf340 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bb9ffb640 1 -- 172.21.15.156:0/450015652 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ba4003520 con 0x7f9bbc0cf340 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 -- 172.21.15.156:0/450015652 >> v2:172.21.15.203:3300/0 conn(0x7f9bbc0b4310 msgr2=0x7f9bbc0cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 --2- 172.21.15.156:0/450015652 >> v2:172.21.15.203:3300/0 conn(0x7f9bbc0b4310 0x7f9bbc0cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 -- 172.21.15.156:0/450015652 >> v1:172.21.15.203:6789/0 conn(0x7f9bbc0cf340 legacy=0x7f9bbc06f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.555+0000 7f9bbaffd640 1 -- 172.21.15.156:0/450015652 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bbc15d0d0 con 0x7f9bbc145c70 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.556+0000 7f9bbaffd640 1 --2- 172.21.15.156:0/450015652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 0x7f9bbc0b3dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009870 tx=0x7f9bac02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3811c514ccfd73e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.557+0000 7f9bb9ffb640 1 -- 172.21.15.156:0/450015652 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac02e070 con 0x7f9bbc145c70 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.557+0000 7f9bb9ffb640 1 -- 172.21.15.156:0/450015652 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bac037070 con 0x7f9bbc145c70 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.558+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 msgr2=0x7f9bbc0b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.558+0000 7f9bb9ffb640 1 -- 172.21.15.156:0/450015652 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac02e070 con 0x7f9bbc145c70 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.558+0000 7f9bc1360640 1 --2- 172.21.15.156:0/450015652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 0x7f9bbc0b3dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009870 tx=0x7f9bac02b880 comp rx=0 tx=0).stop 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.559+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 shutdown_connections 2024-03-29T04:09:01.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.559+0000 7f9bc1360640 1 --2- 172.21.15.156:0/450015652 >> v2:172.21.15.203:3300/0 conn(0x7f9bbc0b4310 0x7f9bbc0cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.559+0000 7f9bc1360640 1 --2- 172.21.15.156:0/450015652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc145c70 0x7f9bbc0b3dd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.559+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 >> 172.21.15.156:0/450015652 conn(0x7f9bbc0b7660 msgr2=0x7f9bbc0b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.559+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 shutdown_connections 2024-03-29T04:09:01.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.559+0000 7f9bc1360640 1 -- 172.21.15.156:0/450015652 wait complete. 2024-03-29T04:09:01.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.561+0000 7f9bc1360640 1 Processor -- start 2024-03-29T04:09:01.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.561+0000 7f9bc1360640 1 -- start start 2024-03-29T04:09:01.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bc1360640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bbc0b4310 0x7f9bbc1f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2918059810 con 0x7f291c0b3d80 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bc1360640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 0x7f9bbc1f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bc1360640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bbc145c70 0x7f9bbc13e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bc1360640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9bbc0cf930 con 0x7f9bbc0b4310 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bc1360640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9bbc0cf7b0 con 0x7f9bbc145c70 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bc1360640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9bbc146050 con 0x7f9bbc0cf340 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bba7fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 0x7f9bbc1f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bba7fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 0x7f9bbc1f94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:48174/0 (socket says 172.21.15.156:48174) 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bba7fc640 1 -- 172.21.15.156:0/3125559339 learned_addr learned my addr 172.21.15.156:0/3125559339 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bbb7fe640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bbc145c70 0x7f9bbc13e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bbaffd640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bbc0b4310 0x7f9bbc1f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.562+0000 7f9bba7fc640 1 -- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bbc145c70 msgr2=0x7f9bbc13e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bba7fc640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bbc145c70 0x7f9bbc13e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bba7fc640 1 -- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bbc0b4310 msgr2=0x7f9bbc1f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bba7fc640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bbc0b4310 0x7f9bbc1f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bba7fc640 1 -- 172.21.15.156:0/3125559339 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bbc1f9a30 con 0x7f9bbc0cf340 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bbb7fe640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bbc145c70 0x7f9bbc13e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bbaffd640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bbc0b4310 0x7f9bbc1f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bba7fc640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 0x7f9bbc1f94f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009660 tx=0x7f9bb0009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb001c070 con 0x7f9bbc0cf340 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bb000c6d0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb000cac0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bbc06d8b0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.563+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bbc06de70 con 0x7f9bbc0cf340 2024-03-29T04:09:01.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.564+0000 7f833b59e640 1 Processor -- start 2024-03-29T04:09:01.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.564+0000 7f833b59e640 1 -- start start 2024-03-29T04:09:01.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.565+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9bb001f070 con 0x7f9bbc0cf340 2024-03-29T04:09:01.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.565+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bbc0b7db0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f833b59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83340e6920 0x7f8334056370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f833a59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83340e6920 0x7f8334056370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f833a59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83340e6920 0x7f8334056370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48474/0 (socket says 172.21.15.156:48474) 2024-03-29T04:09:01.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f8339d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f83340e6d00 0x7f8334054bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47508/0 (socket says 172.21.15.156:47508) 2024-03-29T04:09:01.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f9ba37fe640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ba806c600 0x7f9ba806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9bb008ae00 con 0x7f9bbc0cf340 2024-03-29T04:09:01.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f9bbaffd640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ba806c600 0x7f9ba806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.566+0000 7f833a59c640 1 -- 172.21.15.156:0/1324506071 learned_addr learned my addr 172.21.15.156:0/1324506071 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8334006b40 con 0x7f83340e6d00 2024-03-29T04:09:01.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8334006e20 con 0x7f83340552d0 2024-03-29T04:09:01.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8334007100 con 0x7f83340e6920 2024-03-29T04:09:01.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635098672 0 0) 0x7f8334006b40 con 0x7f83340e6d00 2024-03-29T04:09:01.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f9bbaffd640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ba806c600 0x7f9ba806eac0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bac036000 tx=0x7f9bac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8324003520 con 0x7f83340e6d00 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4135388850 0 0) 0x7f8334006e20 con 0x7f83340552d0 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8334006b40 con 0x7f83340552d0 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833a59c640 1 -- 172.21.15.156:0/1324506071 >> v1:172.21.15.203:6789/0 conn(0x7f83340552d0 legacy=0x7f8334007900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833a59c640 1 -- 172.21.15.156:0/1324506071 >> v1:172.21.15.156:6789/0 conn(0x7f83340e6d00 legacy=0x7f8334054bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.567+0000 7f833a59c640 1 -- 172.21.15.156:0/1324506071 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83340073e0 con 0x7f83340e6920 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.568+0000 7f833a59c640 1 --2- 172.21.15.156:0/1324506071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f8334056370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f833c050b60 tx=0x7f833c0a7090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445373aef4534a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.569+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0b4070 con 0x7f83340e6920 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.569+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833c0af440 con 0x7f83340e6920 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.569+0000 7f833959a640 1 -- 172.21.15.156:0/1324506071 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0af7f0 con 0x7f83340e6920 2024-03-29T04:09:01.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 msgr2=0x7f8334056370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 --2- 172.21.15.156:0/1324506071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f8334056370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f833c050b60 tx=0x7f833c0a7090 comp rx=0 tx=0).stop 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 shutdown_connections 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 --2- 172.21.15.156:0/1324506071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f8334056370 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 >> 172.21.15.156:0/1324506071 conn(0x7f833401ef30 msgr2=0x7f8334055c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 shutdown_connections 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.570+0000 7f833b59e640 1 -- 172.21.15.156:0/1324506071 wait complete. 2024-03-29T04:09:01.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.571+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9bb00538e0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.573+0000 7f833b59e640 1 Processor -- start 2024-03-29T04:09:01.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.573+0000 7f833b59e640 1 -- start start 2024-03-29T04:09:01.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.574+0000 7f833b59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 0x7f833404f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.574+0000 7f833b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f83340e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833a59c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 0x7f833404f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f8339d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f83340e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833a59c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 0x7f833404f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:48188/0 (socket says 172.21.15.156:48188) 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f8339d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f83340e3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48480/0 (socket says 172.21.15.156:48480) 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833b59e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83340e6d00 0x7f83340e4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833a59c640 1 -- 172.21.15.156:0/4233075388 learned_addr learned my addr 172.21.15.156:0/4233075388 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83340558d0 con 0x7f83340e6920 2024-03-29T04:09:01.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f833403b790 con 0x7f83340e6d00 2024-03-29T04:09:01.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8334055a50 con 0x7f83340552d0 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.575+0000 7f833ad9d640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83340e6d00 0x7f83340e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833a59c640 1 -- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83340e6d00 msgr2=0x7f83340e4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833a59c640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83340e6d00 0x7f83340e4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833a59c640 1 -- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 msgr2=0x7f83340e3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833a59c640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f83340e3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833a59c640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833404a9e0 con 0x7f83340552d0 2024-03-29T04:09:01.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f8339d9b640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f83340e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833a59c640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 0x7f833404f080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f833c074450 tx=0x7f833c06b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0bc070 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833c0ae850 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0b4040 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833404acc0 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.576+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f833404cd30 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.578+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83340057d0 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.579+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f833c0bf070 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.579+0000 7f832affd640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f831c06c530 0x7f831c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.579+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f833c1287c0 con 0x7f83340552d0 2024-03-29T04:09:01.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.580+0000 7f8339d9b640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f831c06c530 0x7f831c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.580+0000 7f8339d9b640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f831c06c530 0x7f831c06e9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f833404b470 tx=0x7f8330007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.584+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f833c0f6780 con 0x7f83340552d0 2024-03-29T04:09:01.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.686+0000 7f707b055640 1 Processor -- start 2024-03-29T04:09:01.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.686+0000 7f707b055640 1 -- start start 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.687+0000 7f707b055640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7074145940 0x7f70741489c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.687+0000 7f707b055640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7074162eb0 0x7f7074145f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f706bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7074162eb0 0x7f7074145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f706bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7074162eb0 0x7f7074145f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48486/0 (socket says 172.21.15.156:48486) 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f70795f2640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f70740b3dd0 0x7f70741669c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47512/0 (socket says 172.21.15.156:47512) 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f7078df1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7074145940 0x7f70741489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f7078df1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7074145940 0x7f70741489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:48198/0 (socket says 172.21.15.156:48198) 2024-03-29T04:09:01.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f707b055640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7074163470 con 0x7f70740b3dd0 2024-03-29T04:09:01.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.688+0000 7f706bfff640 1 -- 172.21.15.156:0/956458941 learned_addr learned my addr 172.21.15.156:0/956458941 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f70740b5470 con 0x7f7074162eb0 2024-03-29T04:09:01.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f706b7fe640 1 -- 172.21.15.156:0/956458941 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876538407 0 0) 0x7f7074163470 con 0x7f70740b3dd0 2024-03-29T04:09:01.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7074148f00 con 0x7f7074145940 2024-03-29T04:09:01.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f706b7fe640 1 -- 172.21.15.156:0/956458941 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f705c003520 con 0x7f70740b3dd0 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f706bfff640 1 -- 172.21.15.156:0/956458941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7074145940 msgr2=0x7f70741489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f706bfff640 1 --2- 172.21.15.156:0/956458941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7074145940 0x7f70741489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f706bfff640 1 -- 172.21.15.156:0/956458941 >> v1:172.21.15.156:6789/0 conn(0x7f70740b3dd0 legacy=0x7f70741669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.689+0000 7f706bfff640 1 -- 172.21.15.156:0/956458941 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7074163750 con 0x7f7074162eb0 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.690+0000 7f706bfff640 1 --2- 172.21.15.156:0/956458941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074162eb0 0x7f7074145f20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70640098a0 tx=0x7f706402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10503a1044e78ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.691+0000 7f706b7fe640 1 -- 172.21.15.156:0/956458941 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7064035020 con 0x7f7074162eb0 2024-03-29T04:09:01.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.691+0000 7f706b7fe640 1 -- 172.21.15.156:0/956458941 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7064031870 con 0x7f7074162eb0 2024-03-29T04:09:01.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.691+0000 7f706b7fe640 1 -- 172.21.15.156:0/956458941 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7064031c80 con 0x7f7074162eb0 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074162eb0 msgr2=0x7f7074145f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 --2- 172.21.15.156:0/956458941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074162eb0 0x7f7074145f20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70640098a0 tx=0x7f706402b910 comp rx=0 tx=0).stop 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 shutdown_connections 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 --2- 172.21.15.156:0/956458941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074162eb0 0x7f7074145f20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 --2- 172.21.15.156:0/956458941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7074145940 0x7f70741489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 >> 172.21.15.156:0/956458941 conn(0x7f70740b7660 msgr2=0x7f70740b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 shutdown_connections 2024-03-29T04:09:01.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.692+0000 7f707b055640 1 -- 172.21.15.156:0/956458941 wait complete. 2024-03-29T04:09:01.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.694+0000 7f707b055640 1 Processor -- start 2024-03-29T04:09:01.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.695+0000 7f707b055640 1 -- start start 2024-03-29T04:09:01.675 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.696+0000 7f707b055640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 0x7f7074148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.696+0000 7f707b055640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 0x7f70741403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.696+0000 7f706bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 0x7f70741403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f7078df1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 0x7f7074148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f7078df1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 0x7f7074148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:48214/0 (socket says 172.21.15.156:48214) 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f706bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 0x7f70741403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48502/0 (socket says 172.21.15.156:48502) 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f7078df1640 1 -- 172.21.15.156:0/3655340937 learned_addr learned my addr 172.21.15.156:0/3655340937 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f707b055640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7074162eb0 0x7f7074143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f707b055640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70741664a0 con 0x7f7074145940 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f707b055640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7074163d10 con 0x7f7074162eb0 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f707b055640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7074146460 con 0x7f70740b3dd0 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.697+0000 7f70795f2640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7074162eb0 0x7f7074143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f7078df1640 1 -- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7074162eb0 msgr2=0x7f7074143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f7078df1640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7074162eb0 0x7f7074143920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f7078df1640 1 -- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 msgr2=0x7f70741403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f7078df1640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 0x7f70741403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f7078df1640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7074148cb0 con 0x7f70740b3dd0 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f706bfff640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 0x7f70741403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f7078df1640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 0x7f7074148770 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9b0 tx=0x7f707000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.698+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070017070 con 0x7f70740b3dd0 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.699+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707000c9a0 con 0x7f70740b3dd0 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.699+0000 7f707b055640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7074140da0 con 0x7f70740b3dd0 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.699+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707000cd90 con 0x7f70740b3dd0 2024-03-29T04:09:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.699+0000 7f707b055640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f707406cb70 con 0x7f70740b3dd0 2024-03-29T04:09:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.700+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70741412c0 con 0x7f70740b3dd0 2024-03-29T04:09:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.701+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f707001a070 con 0x7f70740b3dd0 2024-03-29T04:09:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.701+0000 7f70697fa640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f706006c530 0x7f706006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.701+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f707008b5d0 con 0x7f70740b3dd0 2024-03-29T04:09:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.701+0000 7f706bfff640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f706006c530 0x7f706006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.702+0000 7f706bfff640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f706006c530 0x7f706006e9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7064036000 tx=0x7f706403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:01.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.706+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f70741412c0 con 0x7f70740b3dd0 2024-03-29T04:09:01.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.788+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f413404f9d0 con 0x7f413415cb50 2024-03-29T04:09:01.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.789+0000 7f412b7fe640 1 -- 172.21.15.156:0/1016327848 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f411c07b720 con 0x7f413415cb50 2024-03-29T04:09:01.768 INFO:teuthology.orchestra.run.smithi156.stdout:55834574853 2024-03-29T04:09:01.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.791+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f411406c530 msgr2=0x7f411406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.791+0000 7f413964a640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f411406c530 0x7f411406e9f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4124007fd0 tx=0x7f4124009040 comp rx=0 tx=0).stop 2024-03-29T04:09:01.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.791+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f413415cb50 msgr2=0x7f413415afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.791+0000 7f413964a640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f413415cb50 0x7f413415afe0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41340734f0 tx=0x7f411c0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.792+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 shutdown_connections 2024-03-29T04:09:01.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.792+0000 7f413964a640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f411406c530 0x7f411406e9f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.792+0000 7f413964a640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41341584d0 0x7f41340b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.792+0000 7f413964a640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f413415cb50 0x7f413415afe0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.792+0000 7f413964a640 1 --2- 172.21.15.156:0/1016327848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41340cddc0 0x7f413415c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.792+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 >> 172.21.15.156:0/1016327848 conn(0x7f41340b3d60 msgr2=0x7f4134159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.794+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 shutdown_connections 2024-03-29T04:09:01.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.794+0000 7f413964a640 1 -- 172.21.15.156:0/1016327848 wait complete. 2024-03-29T04:09:01.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-03-29T04:09:01.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.849+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f291c06c200 con 0x7f291c0b3d80 2024-03-29T04:09:01.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.849+0000 7f29127fc640 1 -- 172.21.15.156:0/2375512648 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2918059110 con 0x7f291c0b3d80 2024-03-29T04:09:01.829 INFO:teuthology.orchestra.run.smithi156.stdout:60129542150 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f290006c630 msgr2=0x7f290006eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f290006c630 0x7f290006eaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f290c036000 tx=0x7f290c03c040 comp rx=0 tx=0).stop 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 msgr2=0x7f291c06b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 0x7f291c06b5b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f291800c7b0 tx=0x7f291800cc70 comp rx=0 tx=0).stop 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 shutdown_connections 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f290006c630 0x7f290006eaf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f291c0d0280 0x7f291c06baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291c0b3d80 0x7f291c06b5b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 --2- 172.21.15.156:0/2375512648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f291c072730 0x7f291c161b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 >> 172.21.15.156:0/2375512648 conn(0x7f291c0b1f30 msgr2=0x7f291c0ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 shutdown_connections 2024-03-29T04:09:01.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.852+0000 7f2923ea7640 1 -- 172.21.15.156:0/2375512648 wait complete. 2024-03-29T04:09:01.847 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2024-03-29T04:09:01.847 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:01.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.919+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8334052ff0 con 0x7f83340552d0 2024-03-29T04:09:01.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.920+0000 7f832affd640 1 -- 172.21.15.156:0/4233075388 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f833c0faa80 con 0x7f83340552d0 2024-03-29T04:09:01.901 INFO:teuthology.orchestra.run.smithi156.stdout:60129542151 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f831c06c530 msgr2=0x7f831c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f831c06c530 0x7f831c06e9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f833404b470 tx=0x7f8330007d30 comp rx=0 tx=0).stop 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 msgr2=0x7f833404f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 0x7f833404f080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f833c074450 tx=0x7f833c06b3a0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 shutdown_connections 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f831c06c530 0x7f831c06e9f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83340e6d00 0x7f83340e4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83340e6920 0x7f83340e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 --2- 172.21.15.156:0/4233075388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83340552d0 0x7f833404f080 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 >> 172.21.15.156:0/4233075388 conn(0x7f833401ef30 msgr2=0x7f83340e5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 shutdown_connections 2024-03-29T04:09:01.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.923+0000 7f833b59e640 1 -- 172.21.15.156:0/4233075388 wait complete. 2024-03-29T04:09:01.914 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2024-03-29T04:09:01.914 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.947+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8584005850 con 0x7f85a809c020 2024-03-29T04:09:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.947+0000 7f859dffb640 1 -- 172.21.15.156:0/195726090 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85a007ecf0 con 0x7f85a809c020 2024-03-29T04:09:01.927 INFO:teuthology.orchestra.run.smithi156.stdout:55834574855 2024-03-29T04:09:01.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f858c06c630 msgr2=0x7f858c06eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f858c06c630 0x7f858c06eaf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85a8147250 tx=0x7f85a4009320 comp rx=0 tx=0).stop 2024-03-29T04:09:01.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85a809c020 msgr2=0x7f85a81f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85a809c020 0x7f85a81f35c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f85a80b47b0 tx=0x7f85a0001650 comp rx=0 tx=0).stop 2024-03-29T04:09:01.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 shutdown_connections 2024-03-29T04:09:01.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f858c06c630 0x7f858c06eaf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85a809c020 0x7f85a81f35c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85a81f4240 0x7f85a815e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 --2- 172.21.15.156:0/195726090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85940051c0 0x7f85a81396d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 >> 172.21.15.156:0/195726090 conn(0x7f85a80b65f0 msgr2=0x7f85a80ae0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 shutdown_connections 2024-03-29T04:09:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.949+0000 7f858b7fe640 1 -- 172.21.15.156:0/195726090 wait complete. 2024-03-29T04:09:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.954+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9bbc04f9d0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.954+0000 7f9ba37fe640 1 -- 172.21.15.156:0/3125559339 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9bbc0b7db0 con 0x7f9bbc0cf340 2024-03-29T04:09:01.934 INFO:teuthology.orchestra.run.smithi156.stdout:55834574855 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ba806c600 msgr2=0x7f9ba806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ba806c600 0x7f9ba806eac0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bac036000 tx=0x7f9bac008030 comp rx=0 tx=0).stop 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 msgr2=0x7f9bbc1f94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 0x7f9bbc1f94f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009660 tx=0x7f9bb0009b20 comp rx=0 tx=0).stop 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 shutdown_connections 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ba806c600 0x7f9ba806eac0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bbc145c70 0x7f9bbc13e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9bbc0cf340 0x7f9bbc1f94f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 --2- 172.21.15.156:0/3125559339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bbc0b4310 0x7f9bbc1f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 >> 172.21.15.156:0/3125559339 conn(0x7f9bbc0b7660 msgr2=0x7f9bbc0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 shutdown_connections 2024-03-29T04:09:01.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.957+0000 7f9bc1360640 1 -- 172.21.15.156:0/3125559339 wait complete. 2024-03-29T04:09:01.943 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.0 2024-03-29T04:09:01.943 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:01.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2024-03-29T04:09:01.948 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:01.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.981+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f70741641f0 con 0x7f70740b3dd0 2024-03-29T04:09:01.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.982+0000 7f70697fa640 1 -- 172.21.15.156:0/3655340937 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7070059630 con 0x7f70740b3dd0 2024-03-29T04:09:01.961 INFO:teuthology.orchestra.run.smithi156.stdout:60129542150 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f706006c530 msgr2=0x7f706006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f706006c530 0x7f706006e9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7064036000 tx=0x7f706403c040 comp rx=0 tx=0).stop 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 msgr2=0x7f7074148770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 0x7f7074148770 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9b0 tx=0x7f707000ee70 comp rx=0 tx=0).stop 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 shutdown_connections 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f706006c530 0x7f706006e9f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7074162eb0 0x7f7074143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7074145940 0x7f70741403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 --2- 172.21.15.156:0/3655340937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70740b3dd0 0x7f7074148770 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:01.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 >> 172.21.15.156:0/3655340937 conn(0x7f70740b7660 msgr2=0x7f70740b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:01.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 shutdown_connections 2024-03-29T04:09:01.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:01.984+0000 7f705affd640 1 -- 172.21.15.156:0/3655340937 wait complete. 2024-03-29T04:09:01.974 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2024-03-29T04:09:01.974 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:02.786 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T04:09:02.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.912+0000 7f5023ce6640 1 Processor -- start 2024-03-29T04:09:02.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.912+0000 7f5023ce6640 1 -- start start 2024-03-29T04:09:02.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.913+0000 7f5023ce6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f501c144650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.913+0000 7f5023ce6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c162130 con 0x7f501c0b3dd0 2024-03-29T04:09:02.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.913+0000 7f5023ce6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c1422c0 con 0x7f501c1415d0 2024-03-29T04:09:02.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.913+0000 7f5023ce6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f501c144b90 con 0x7f501c15eaa0 2024-03-29T04:09:02.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5022283640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f501c0b3dd0 0x7f501c1625e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47518/0 (socket says 172.21.15.156:47518) 2024-03-29T04:09:02.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5022283640 1 -- 172.21.15.156:0/1859155725 learned_addr learned my addr 172.21.15.156:0/1859155725 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021281640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f501c144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042803960 0 0) 0x7f501c162130 con 0x7f501c0b3dd0 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5008003520 con 0x7f501c0b3dd0 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 537372751 0 0) 0x7f5008003520 con 0x7f501c0b3dd0 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 msgr2=0x7f501c144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f501c144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 >> v1:172.21.15.203:6789/0 conn(0x7f501c1415d0 legacy=0x7f501c141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c15f0f0 con 0x7f501c0b3dd0 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f50140019f0 con 0x7f501c0b3dd0 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 >> v1:172.21.15.156:6789/0 conn(0x7f501c0b3dd0 legacy=0x7f501c1625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f5008003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008004390 0x7f5008004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f50080050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f501c144b90 con 0x7f50080039e0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f500800b9f0 con 0x7f5008004390 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f500800bbf0 con 0x7f501c15eaa0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021a82640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f5008003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021281640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008004390 0x7f5008004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5022283640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f50080050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021a82640 1 -- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 msgr2=0x7f50080050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021a82640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f50080050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021a82640 1 -- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008004390 msgr2=0x7f5008004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021a82640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008004390 0x7f5008004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5021a82640 1 -- 172.21.15.156:0/1859155725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5008003520 con 0x7f50080039e0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.914+0000 7f5022283640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c15eaa0 0x7f50080050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.915+0000 7f5021a82640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f5008003dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5018001860 tx=0x7f5008005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5f8ab42399e963 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:02.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.915+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501802f050 con 0x7f50080039e0 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.915+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5018033650 con 0x7f50080039e0 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.915+0000 7f5020a80640 1 -- 172.21.15.156:0/1859155725 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018001430 con 0x7f50080039e0 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5023ce6640 1 -- 172.21.15.156:0/1859155725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 msgr2=0x7f5008003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5023ce6640 1 --2- 172.21.15.156:0/1859155725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f5008003dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5018001860 tx=0x7f5008005830 comp rx=0 tx=0).stop 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5022283640 1 -- 172.21.15.156:0/1859155725 reap_dead start 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5023ce6640 1 -- 172.21.15.156:0/1859155725 shutdown_connections 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5023ce6640 1 -- 172.21.15.156:0/1859155725 >> 172.21.15.156:0/1859155725 conn(0x7f501c0b7660 msgr2=0x7f501c0b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5023ce6640 1 -- 172.21.15.156:0/1859155725 shutdown_connections 2024-03-29T04:09:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.916+0000 7f5023ce6640 1 -- 172.21.15.156:0/1859155725 wait complete. 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.917+0000 7f5023ce6640 1 Processor -- start 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.917+0000 7f5023ce6640 1 -- start start 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5023ce6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f501c140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5023ce6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f501c0cc510 0x7f501c06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5023ce6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c0cdff0 0x7f501c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5023ce6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f501c15f990 con 0x7f50080039e0 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5023ce6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f501c15f6b0 con 0x7f501c0cc510 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5023ce6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f501c162cf0 con 0x7f501c0cdff0 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f501c140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f501c140d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48514/0 (socket says 172.21.15.156:48514) 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 -- 172.21.15.156:0/617846180 learned_addr learned my addr 172.21.15.156:0/617846180 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5022283640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c0cdff0 0x7f501c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021281640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f501c0cc510 0x7f501c06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 -- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c0cdff0 msgr2=0x7f501c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c0cdff0 0x7f501c06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 -- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f501c0cc510 msgr2=0x7f501c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f501c0cc510 0x7f501c06c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5021a82640 1 -- 172.21.15.156:0/617846180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c0d0290 con 0x7f50080039e0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.918+0000 7f5022283640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c0cdff0 0x7f501c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.919+0000 7f5021a82640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f501c140d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5018001980 tx=0x7f5018001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.919+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50180035f0 con 0x7f50080039e0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.919+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5018003790 con 0x7f50080039e0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.919+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018003b40 con 0x7f50080039e0 2024-03-29T04:09:02.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.919+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501c0cee20 con 0x7f50080039e0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.919+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f501c0cf3e0 con 0x7f50080039e0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.920+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f501c157100 con 0x7f50080039e0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.920+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5018003ce0 con 0x7f50080039e0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.921+0000 7f50127fc640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f500006c530 0x7f500006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.921+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f50180ad810 con 0x7f50080039e0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.921+0000 7f5021281640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f500006c530 0x7f500006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.921+0000 7f5021281640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f500006c530 0x7f500006e9f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f501c155b10 tx=0x7f500c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:02.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:02.925+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f501c157100 con 0x7f50080039e0 2024-03-29T04:09:03.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.153+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f501c0d2f60 con 0x7f50080039e0 2024-03-29T04:09:03.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.154+0000 7f50127fc640 1 -- 172.21.15.156:0/617846180 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f501804a060 con 0x7f50080039e0 2024-03-29T04:09:03.133 INFO:teuthology.orchestra.run.smithi156.stdout:55834574855 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f500006c530 msgr2=0x7f500006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f500006c530 0x7f500006e9f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f501c155b10 tx=0x7f500c008040 comp rx=0 tx=0).stop 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 msgr2=0x7f501c140d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f501c140d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5018001980 tx=0x7f5018001230 comp rx=0 tx=0).stop 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 shutdown_connections 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f500006c530 0x7f500006e9f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f501c0cdff0 0x7f501c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f501c0cc510 0x7f501c06c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 --2- 172.21.15.156:0/617846180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50080039e0 0x7f501c140d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 >> 172.21.15.156:0/617846180 conn(0x7f501c0b7660 msgr2=0x7f501c0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:03.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 shutdown_connections 2024-03-29T04:09:03.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.156+0000 7f5023ce6640 1 -- 172.21.15.156:0/617846180 wait complete. 2024-03-29T04:09:03.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.1 2024-03-29T04:09:03.146 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:03.146 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-29T04:09:03.146 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:09:03.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.268+0000 7f1d29350640 1 Processor -- start 2024-03-29T04:09:03.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.268+0000 7f1d29350640 1 -- start start 2024-03-29T04:09:03.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d29350640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d2415c4f0 0x7f1d24160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d29350640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2406fc80 con 0x7f1d24162230 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d29350640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d241627a0 con 0x7f1d24070ad0 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d29350640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1d24162a80 con 0x7f1d2415c4f0 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d227fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1d24162230 0x7f1d240703c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47532/0 (socket says 172.21.15.156:47532) 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d227fc640 1 -- 172.21.15.156:0/3491457698 learned_addr learned my addr 172.21.15.156:0/3491457698 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d22ffd640 1 --2- 172.21.15.156:0/3491457698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d2415c4f0 0x7f1d24160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1616296038 0 0) 0x7f1d2406fc80 con 0x7f1d24162230 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d0c003520 con 0x7f1d24162230 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466152726 0 0) 0x7f1d241627a0 con 0x7f1d24070ad0 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.269+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d2406fc80 con 0x7f1d24070ad0 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.270+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 719206637 0 0) 0x7f1d0c003520 con 0x7f1d24162230 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.270+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3491457698 >> v1:172.21.15.203:6789/0 conn(0x7f1d24070ad0 legacy=0x7f1d240b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.270+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3491457698 >> v1:172.21.15.156:6789/0 conn(0x7f1d24162230 legacy=0x7f1d240703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.270+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3491457698 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d24162d60 con 0x7f1d2415c4f0 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.271+0000 7f1d22ffd640 1 --2- 172.21.15.156:0/3491457698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d2415c4f0 0x7f1d24160030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1d140097d0 tx=0x7f1d1402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb4e32f188b011cf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.271+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14037070 con 0x7f1d2415c4f0 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.271+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d14030510 con 0x7f1d2415c4f0 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.271+0000 7f1d21ffb640 1 -- 172.21.15.156:0/3491457698 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18001670 con 0x7f1d2415c4f0 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 -- 172.21.15.156:0/3491457698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d2415c4f0 msgr2=0x7f1d24160030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 --2- 172.21.15.156:0/3491457698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d2415c4f0 0x7f1d24160030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1d140097d0 tx=0x7f1d1402b800 comp rx=0 tx=0).stop 2024-03-29T04:09:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 -- 172.21.15.156:0/3491457698 shutdown_connections 2024-03-29T04:09:03.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 --2- 172.21.15.156:0/3491457698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d2415c4f0 0x7f1d24160030 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 -- 172.21.15.156:0/3491457698 >> 172.21.15.156:0/3491457698 conn(0x7f1d240d2250 msgr2=0x7f1d240d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:03.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 -- 172.21.15.156:0/3491457698 shutdown_connections 2024-03-29T04:09:03.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.272+0000 7f1d29350640 1 -- 172.21.15.156:0/3491457698 wait complete. 2024-03-29T04:09:03.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.274+0000 7f1d29350640 1 Processor -- start 2024-03-29T04:09:03.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.274+0000 7f1d29350640 1 -- start start 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d29350640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 0x7f1d241f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d29350640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d2415c4f0 0x7f1d240b7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d29350640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d24162230 0x7f1d240b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d29350640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d240d05a0 con 0x7f1d24070ad0 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d29350640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d240d0420 con 0x7f1d2415c4f0 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d29350640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1d240b1870 con 0x7f1d24162230 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 0x7f1d241f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 0x7f1d241f45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48520/0 (socket says 172.21.15.156:48520) 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3732184561 learned_addr learned my addr 172.21.15.156:0/3732184561 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d227fc640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d2415c4f0 0x7f1d240b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d237fe640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d24162230 0x7f1d240b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d24162230 msgr2=0x7f1d240b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d24162230 0x7f1d240b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d2415c4f0 msgr2=0x7f1d240b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d2415c4f0 0x7f1d240b7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d22ffd640 1 -- 172.21.15.156:0/3732184561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d240b6890 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d227fc640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d2415c4f0 0x7f1d240b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.275+0000 7f1d237fe640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d24162230 0x7f1d240b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.276+0000 7f1d22ffd640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 0x7f1d241f45f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d1402b840 tx=0x7f1d140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.276+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14041020 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.276+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1403c870 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.276+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d240b6b70 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.276+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14037040 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.276+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d240b7130 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.277+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d24068610 con 0x7f1d24070ad0 2024-03-29T04:09:03.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.277+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1d14044020 con 0x7f1d24070ad0 2024-03-29T04:09:03.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.278+0000 7f1d0b7fe640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d1006c530 0x7f1d1006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.278+0000 7f1d227fc640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d1006c530 0x7f1d1006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.278+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1d140ad160 con 0x7f1d24070ad0 2024-03-29T04:09:03.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.278+0000 7f1d227fc640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d1006c530 0x7f1d1006e9f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1d2414b040 tx=0x7f1d18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.286+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1d1404b030 con 0x7f1d24070ad0 2024-03-29T04:09:03.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.463+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1d240d3160 con 0x7f1d1006c530 2024-03-29T04:09:03.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.469+0000 7f1d0b7fe640 1 -- 172.21.15.156:0/3732184561 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f1d240d3160 con 0x7f1d1006c530 2024-03-29T04:09:03.451 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:03.453 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d1006c530 msgr2=0x7f1d1006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d1006c530 0x7f1d1006e9f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1d2414b040 tx=0x7f1d18009040 comp rx=0 tx=0).stop 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 msgr2=0x7f1d241f45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 0x7f1d241f45f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d1402b840 tx=0x7f1d140046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 shutdown_connections 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d1006c530 0x7f1d1006e9f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1d24162230 0x7f1d240b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d2415c4f0 0x7f1d240b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 --2- 172.21.15.156:0/3732184561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d24070ad0 0x7f1d241f45f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 >> 172.21.15.156:0/3732184561 conn(0x7f1d240d2250 msgr2=0x7f1d2415a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:03.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 shutdown_connections 2024-03-29T04:09:03.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.472+0000 7f1d29350640 1 -- 172.21.15.156:0/3732184561 wait complete. 2024-03-29T04:09:03.463 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T04:09:01.638753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235692,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739028,"statfs":{"total":773094113280,"available":772852764672,"internally_reserved":0,"allocated":2957312,"data_stored":1781436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001523"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546238+0000","last_change":"2024-03-29T04:08:53.827540+0000","last_active":"2024-03-29T04:08:55.546238+0000","last_peered":"2024-03-29T04:08:55.546238+0000","last_clean":"2024-03-29T04:08:55.546238+0000","last_became_active":"2024-03-29T04:08:53.827301+0000","last_became_peered":"2024-03-29T04:08:53.827301+0000","last_unstale":"2024-03-29T04:08:55.546238+0000","last_undegraded":"2024-03-29T04:08:55.546238+0000","last_fullsized":"2024-03-29T04:08:55.546238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812222+0000","last_change":"2024-03-29T04:08:53.809513+0000","last_active":"2024-03-29T04:08:54.812222+0000","last_peered":"2024-03-29T04:08:54.812222+0000","last_clean":"2024-03-29T04:08:54.812222+0000","last_became_active":"2024-03-29T04:08:53.809236+0000","last_became_peered":"2024-03-29T04:08:53.809236+0000","last_unstale":"2024-03-29T04:08:54.812222+0000","last_undegraded":"2024-03-29T04:08:54.812222+0000","last_fullsized":"2024-03-29T04:08:54.812222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924239+0000","last_change":"2024-03-29T04:08:53.821233+0000","last_active":"2024-03-29T04:08:55.924239+0000","last_peered":"2024-03-29T04:08:55.924239+0000","last_clean":"2024-03-29T04:08:55.924239+0000","last_became_active":"2024-03-29T04:08:53.820557+0000","last_became_peered":"2024-03-29T04:08:53.820557+0000","last_unstale":"2024-03-29T04:08:55.924239+0000","last_undegraded":"2024-03-29T04:08:55.924239+0000","last_fullsized":"2024-03-29T04:08:55.924239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814954+0000","last_change":"2024-03-29T04:08:53.824851+0000","last_active":"2024-03-29T04:08:54.814954+0000","last_peered":"2024-03-29T04:08:54.814954+0000","last_clean":"2024-03-29T04:08:54.814954+0000","last_became_active":"2024-03-29T04:08:53.824563+0000","last_became_peered":"2024-03-29T04:08:53.824563+0000","last_unstale":"2024-03-29T04:08:54.814954+0000","last_undegraded":"2024-03-29T04:08:54.814954+0000","last_fullsized":"2024-03-29T04:08:54.814954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546094+0000","last_change":"2024-03-29T04:08:53.820567+0000","last_active":"2024-03-29T04:08:55.546094+0000","last_peered":"2024-03-29T04:08:55.546094+0000","last_clean":"2024-03-29T04:08:55.546094+0000","last_became_active":"2024-03-29T04:08:53.820290+0000","last_became_peered":"2024-03-29T04:08:53.820290+0000","last_unstale":"2024-03-29T04:08:55.546094+0000","last_undegraded":"2024-03-29T04:08:55.546094+0000","last_fullsized":"2024-03-29T04:08:55.546094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814434+0000","last_change":"2024-03-29T04:08:53.818966+0000","last_active":"2024-03-29T04:08:54.814434+0000","last_peered":"2024-03-29T04:08:54.814434+0000","last_clean":"2024-03-29T04:08:54.814434+0000","last_became_active":"2024-03-29T04:08:53.818396+0000","last_became_peered":"2024-03-29T04:08:53.818396+0000","last_unstale":"2024-03-29T04:08:54.814434+0000","last_undegraded":"2024-03-29T04:08:54.814434+0000","last_fullsized":"2024-03-29T04:08:54.814434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545835+0000","last_change":"2024-03-29T04:08:53.817600+0000","last_active":"2024-03-29T04:08:55.545835+0000","last_peered":"2024-03-29T04:08:55.545835+0000","last_clean":"2024-03-29T04:08:55.545835+0000","last_became_active":"2024-03-29T04:08:53.817308+0000","last_became_peered":"2024-03-29T04:08:53.817308+0000","last_unstale":"2024-03-29T04:08:55.545835+0000","last_undegraded":"2024-03-29T04:08:55.545835+0000","last_fullsized":"2024-03-29T04:08:55.545835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924368+0000","last_change":"2024-03-29T04:08:53.826247+0000","last_active":"2024-03-29T04:08:55.924368+0000","last_peered":"2024-03-29T04:08:55.924368+0000","last_clean":"2024-03-29T04:08:55.924368+0000","last_became_active":"2024-03-29T04:08:53.825974+0000","last_became_peered":"2024-03-29T04:08:53.825974+0000","last_unstale":"2024-03-29T04:08:55.924368+0000","last_undegraded":"2024-03-29T04:08:55.924368+0000","last_fullsized":"2024-03-29T04:08:55.924368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546330+0000","last_change":"2024-03-29T04:08:53.824041+0000","last_active":"2024-03-29T04:08:55.546330+0000","last_peered":"2024-03-29T04:08:55.546330+0000","last_clean":"2024-03-29T04:08:55.546330+0000","last_became_active":"2024-03-29T04:08:53.823788+0000","last_became_peered":"2024-03-29T04:08:53.823788+0000","last_unstale":"2024-03-29T04:08:55.546330+0000","last_undegraded":"2024-03-29T04:08:55.546330+0000","last_fullsized":"2024-03-29T04:08:55.546330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816424+0000","last_change":"2024-03-29T04:08:53.830205+0000","last_active":"2024-03-29T04:08:54.816424+0000","last_peered":"2024-03-29T04:08:54.816424+0000","last_clean":"2024-03-29T04:08:54.816424+0000","last_became_active":"2024-03-29T04:08:53.829931+0000","last_became_peered":"2024-03-29T04:08:53.829931+0000","last_unstale":"2024-03-29T04:08:54.816424+0000","last_undegraded":"2024-03-29T04:08:54.816424+0000","last_fullsized":"2024-03-29T04:08:54.816424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816205+0000","last_change":"2024-03-29T04:08:53.821458+0000","last_active":"2024-03-29T04:08:54.816205+0000","last_peered":"2024-03-29T04:08:54.816205+0000","last_clean":"2024-03-29T04:08:54.816205+0000","last_became_active":"2024-03-29T04:08:53.821168+0000","last_became_peered":"2024-03-29T04:08:53.821168+0000","last_unstale":"2024-03-29T04:08:54.816205+0000","last_undegraded":"2024-03-29T04:08:54.816205+0000","last_fullsized":"2024-03-29T04:08:54.816205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814279+0000","last_change":"2024-03-29T04:08:53.827058+0000","last_active":"2024-03-29T04:08:54.814279+0000","last_peered":"2024-03-29T04:08:54.814279+0000","last_clean":"2024-03-29T04:08:54.814279+0000","last_became_active":"2024-03-29T04:08:53.826167+0000","last_became_peered":"2024-03-29T04:08:53.826167+0000","last_unstale":"2024-03-29T04:08:54.814279+0000","last_undegraded":"2024-03-29T04:08:54.814279+0000","last_fullsized":"2024-03-29T04:08:54.814279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925452+0000","last_change":"2024-03-29T04:08:53.830239+0000","last_active":"2024-03-29T04:08:55.925452+0000","last_peered":"2024-03-29T04:08:55.925452+0000","last_clean":"2024-03-29T04:08:55.925452+0000","last_became_active":"2024-03-29T04:08:53.829295+0000","last_became_peered":"2024-03-29T04:08:53.829295+0000","last_unstale":"2024-03-29T04:08:55.925452+0000","last_undegraded":"2024-03-29T04:08:55.925452+0000","last_fullsized":"2024-03-29T04:08:55.925452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924536+0000","last_change":"2024-03-29T04:08:53.819700+0000","last_active":"2024-03-29T04:08:55.924536+0000","last_peered":"2024-03-29T04:08:55.924536+0000","last_clean":"2024-03-29T04:08:55.924536+0000","last_became_active":"2024-03-29T04:08:53.819174+0000","last_became_peered":"2024-03-29T04:08:53.819174+0000","last_unstale":"2024-03-29T04:08:55.924536+0000","last_undegraded":"2024-03-29T04:08:55.924536+0000","last_fullsized":"2024-03-29T04:08:55.924536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811825+0000","last_change":"2024-03-29T04:08:53.812928+0000","last_active":"2024-03-29T04:08:54.811825+0000","last_peered":"2024-03-29T04:08:54.811825+0000","last_clean":"2024-03-29T04:08:54.811825+0000","last_became_active":"2024-03-29T04:08:53.812536+0000","last_became_peered":"2024-03-29T04:08:53.812536+0000","last_unstale":"2024-03-29T04:08:54.811825+0000","last_undegraded":"2024-03-29T04:08:54.811825+0000","last_fullsized":"2024-03-29T04:08:54.811825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814066+0000","last_change":"2024-03-29T04:08:53.823468+0000","last_active":"2024-03-29T04:08:54.814066+0000","last_peered":"2024-03-29T04:08:54.814066+0000","last_clean":"2024-03-29T04:08:54.814066+0000","last_became_active":"2024-03-29T04:08:53.822972+0000","last_became_peered":"2024-03-29T04:08:53.822972+0000","last_unstale":"2024-03-29T04:08:54.814066+0000","last_undegraded":"2024-03-29T04:08:54.814066+0000","last_fullsized":"2024-03-29T04:08:54.814066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815690+0000","last_change":"2024-03-29T04:08:53.816411+0000","last_active":"2024-03-29T04:08:54.815690+0000","last_peered":"2024-03-29T04:08:54.815690+0000","last_clean":"2024-03-29T04:08:54.815690+0000","last_became_active":"2024-03-29T04:08:53.816126+0000","last_became_peered":"2024-03-29T04:08:53.816126+0000","last_unstale":"2024-03-29T04:08:54.815690+0000","last_undegraded":"2024-03-29T04:08:54.815690+0000","last_fullsized":"2024-03-29T04:08:54.815690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924748+0000","last_change":"2024-03-29T04:08:53.824360+0000","last_active":"2024-03-29T04:08:55.924748+0000","last_peered":"2024-03-29T04:08:55.924748+0000","last_clean":"2024-03-29T04:08:55.924748+0000","last_became_active":"2024-03-29T04:08:53.823207+0000","last_became_peered":"2024-03-29T04:08:53.823207+0000","last_unstale":"2024-03-29T04:08:55.924748+0000","last_undegraded":"2024-03-29T04:08:55.924748+0000","last_fullsized":"2024-03-29T04:08:55.924748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546389+0000","last_change":"2024-03-29T04:08:53.822584+0000","last_active":"2024-03-29T04:08:55.546389+0000","last_peered":"2024-03-29T04:08:55.546389+0000","last_clean":"2024-03-29T04:08:55.546389+0000","last_became_active":"2024-03-29T04:08:53.821230+0000","last_became_peered":"2024-03-29T04:08:53.821230+0000","last_unstale":"2024-03-29T04:08:55.546389+0000","last_undegraded":"2024-03-29T04:08:55.546389+0000","last_fullsized":"2024-03-29T04:08:55.546389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816186+0000","last_change":"2024-03-29T04:08:53.828332+0000","last_active":"2024-03-29T04:08:54.816186+0000","last_peered":"2024-03-29T04:08:54.816186+0000","last_clean":"2024-03-29T04:08:54.816186+0000","last_became_active":"2024-03-29T04:08:53.827954+0000","last_became_peered":"2024-03-29T04:08:53.827954+0000","last_unstale":"2024-03-29T04:08:54.816186+0000","last_undegraded":"2024-03-29T04:08:54.816186+0000","last_fullsized":"2024-03-29T04:08:54.816186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924504+0000","last_change":"2024-03-29T04:08:53.821853+0000","last_active":"2024-03-29T04:08:55.924504+0000","last_peered":"2024-03-29T04:08:55.924504+0000","last_clean":"2024-03-29T04:08:55.924504+0000","last_became_active":"2024-03-29T04:08:53.820469+0000","last_became_peered":"2024-03-29T04:08:53.820469+0000","last_unstale":"2024-03-29T04:08:55.924504+0000","last_undegraded":"2024-03-29T04:08:55.924504+0000","last_fullsized":"2024-03-29T04:08:55.924504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814489+0000","last_change":"2024-03-29T04:08:53.818641+0000","last_active":"2024-03-29T04:08:54.814489+0000","last_peered":"2024-03-29T04:08:54.814489+0000","last_clean":"2024-03-29T04:08:54.814489+0000","last_became_active":"2024-03-29T04:08:53.818082+0000","last_became_peered":"2024-03-29T04:08:53.818082+0000","last_unstale":"2024-03-29T04:08:54.814489+0000","last_undegraded":"2024-03-29T04:08:54.814489+0000","last_fullsized":"2024-03-29T04:08:54.814489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925283+0000","last_change":"2024-03-29T04:08:53.816408+0000","last_active":"2024-03-29T04:08:55.925283+0000","last_peered":"2024-03-29T04:08:55.925283+0000","last_clean":"2024-03-29T04:08:55.925283+0000","last_became_active":"2024-03-29T04:08:53.816127+0000","last_became_peered":"2024-03-29T04:08:53.816127+0000","last_unstale":"2024-03-29T04:08:55.925283+0000","last_undegraded":"2024-03-29T04:08:55.925283+0000","last_fullsized":"2024-03-29T04:08:55.925283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814214+0000","last_change":"2024-03-29T04:08:53.820680+0000","last_active":"2024-03-29T04:08:54.814214+0000","last_peered":"2024-03-29T04:08:54.814214+0000","last_clean":"2024-03-29T04:08:54.814214+0000","last_became_active":"2024-03-29T04:08:53.819878+0000","last_became_peered":"2024-03-29T04:08:53.819878+0000","last_unstale":"2024-03-29T04:08:54.814214+0000","last_undegraded":"2024-03-29T04:08:54.814214+0000","last_fullsized":"2024-03-29T04:08:54.814214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925741+0000","last_change":"2024-03-29T04:08:53.816044+0000","last_active":"2024-03-29T04:08:55.925741+0000","last_peered":"2024-03-29T04:08:55.925741+0000","last_clean":"2024-03-29T04:08:55.925741+0000","last_became_active":"2024-03-29T04:08:53.815744+0000","last_became_peered":"2024-03-29T04:08:53.815744+0000","last_unstale":"2024-03-29T04:08:55.925741+0000","last_undegraded":"2024-03-29T04:08:55.925741+0000","last_fullsized":"2024-03-29T04:08:55.925741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816521+0000","last_change":"2024-03-29T04:08:53.830594+0000","last_active":"2024-03-29T04:08:54.816521+0000","last_peered":"2024-03-29T04:08:54.816521+0000","last_clean":"2024-03-29T04:08:54.816521+0000","last_became_active":"2024-03-29T04:08:53.830266+0000","last_became_peered":"2024-03-29T04:08:53.830266+0000","last_unstale":"2024-03-29T04:08:54.816521+0000","last_undegraded":"2024-03-29T04:08:54.816521+0000","last_fullsized":"2024-03-29T04:08:54.816521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812076+0000","last_change":"2024-03-29T04:08:53.817458+0000","last_active":"2024-03-29T04:08:54.812076+0000","last_peered":"2024-03-29T04:08:54.812076+0000","last_clean":"2024-03-29T04:08:54.812076+0000","last_became_active":"2024-03-29T04:08:53.817179+0000","last_became_peered":"2024-03-29T04:08:53.817179+0000","last_unstale":"2024-03-29T04:08:54.812076+0000","last_undegraded":"2024-03-29T04:08:54.812076+0000","last_fullsized":"2024-03-29T04:08:54.812076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925638+0000","last_change":"2024-03-29T04:08:53.819794+0000","last_active":"2024-03-29T04:08:55.925638+0000","last_peered":"2024-03-29T04:08:55.925638+0000","last_clean":"2024-03-29T04:08:55.925638+0000","last_became_active":"2024-03-29T04:08:53.819541+0000","last_became_peered":"2024-03-29T04:08:53.819541+0000","last_unstale":"2024-03-29T04:08:55.925638+0000","last_undegraded":"2024-03-29T04:08:55.925638+0000","last_fullsized":"2024-03-29T04:08:55.925638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924635+0000","last_change":"2024-03-29T04:08:53.823205+0000","last_active":"2024-03-29T04:08:55.924635+0000","last_peered":"2024-03-29T04:08:55.924635+0000","last_clean":"2024-03-29T04:08:55.924635+0000","last_became_active":"2024-03-29T04:08:53.822900+0000","last_became_peered":"2024-03-29T04:08:53.822900+0000","last_unstale":"2024-03-29T04:08:55.924635+0000","last_undegraded":"2024-03-29T04:08:55.924635+0000","last_fullsized":"2024-03-29T04:08:55.924635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814624+0000","last_change":"2024-03-29T04:08:53.821709+0000","last_active":"2024-03-29T04:08:54.814624+0000","last_peered":"2024-03-29T04:08:54.814624+0000","last_clean":"2024-03-29T04:08:54.814624+0000","last_became_active":"2024-03-29T04:08:53.820828+0000","last_became_peered":"2024-03-29T04:08:53.820828+0000","last_unstale":"2024-03-29T04:08:54.814624+0000","last_undegraded":"2024-03-29T04:08:54.814624+0000","last_fullsized":"2024-03-29T04:08:54.814624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924175+0000","last_change":"2024-03-29T04:08:53.820303+0000","last_active":"2024-03-29T04:08:55.924175+0000","last_peered":"2024-03-29T04:08:55.924175+0000","last_clean":"2024-03-29T04:08:55.924175+0000","last_became_active":"2024-03-29T04:08:53.820024+0000","last_became_peered":"2024-03-29T04:08:53.820024+0000","last_unstale":"2024-03-29T04:08:55.924175+0000","last_undegraded":"2024-03-29T04:08:55.924175+0000","last_fullsized":"2024-03-29T04:08:55.924175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815813+0000","last_change":"2024-03-29T04:08:53.824803+0000","last_active":"2024-03-29T04:08:54.815813+0000","last_peered":"2024-03-29T04:08:54.815813+0000","last_clean":"2024-03-29T04:08:54.815813+0000","last_became_active":"2024-03-29T04:08:53.824454+0000","last_became_peered":"2024-03-29T04:08:53.824454+0000","last_unstale":"2024-03-29T04:08:54.815813+0000","last_undegraded":"2024-03-29T04:08:54.815813+0000","last_fullsized":"2024-03-29T04:08:54.815813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813758+0000","last_change":"2024-03-29T04:08:53.819297+0000","last_active":"2024-03-29T04:08:54.813758+0000","last_peered":"2024-03-29T04:08:54.813758+0000","last_clean":"2024-03-29T04:08:54.813758+0000","last_became_active":"2024-03-29T04:08:53.819016+0000","last_became_peered":"2024-03-29T04:08:53.819016+0000","last_unstale":"2024-03-29T04:08:54.813758+0000","last_undegraded":"2024-03-29T04:08:54.813758+0000","last_fullsized":"2024-03-29T04:08:54.813758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925546+0000","last_change":"2024-03-29T04:08:52.815351+0000","last_active":"2024-03-29T04:08:55.925546+0000","last_peered":"2024-03-29T04:08:55.925546+0000","last_clean":"2024-03-29T04:08:55.925546+0000","last_became_active":"2024-03-29T04:08:52.814942+0000","last_became_peered":"2024-03-29T04:08:52.814942+0000","last_unstale":"2024-03-29T04:08:55.925546+0000","last_undegraded":"2024-03-29T04:08:55.925546+0000","last_fullsized":"2024-03-29T04:08:55.925546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924762+0000","last_change":"2024-03-29T04:08:53.820823+0000","last_active":"2024-03-29T04:08:55.924762+0000","last_peered":"2024-03-29T04:08:55.924762+0000","last_clean":"2024-03-29T04:08:55.924762+0000","last_became_active":"2024-03-29T04:08:53.820429+0000","last_became_peered":"2024-03-29T04:08:53.820429+0000","last_unstale":"2024-03-29T04:08:55.924762+0000","last_undegraded":"2024-03-29T04:08:55.924762+0000","last_fullsized":"2024-03-29T04:08:55.924762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815991+0000","last_change":"2024-03-29T04:08:52.812694+0000","last_active":"2024-03-29T04:08:54.815991+0000","last_peered":"2024-03-29T04:08:54.815991+0000","last_clean":"2024-03-29T04:08:54.815991+0000","last_became_active":"2024-03-29T04:08:52.812429+0000","last_became_peered":"2024-03-29T04:08:52.812429+0000","last_unstale":"2024-03-29T04:08:54.815991+0000","last_undegraded":"2024-03-29T04:08:54.815991+0000","last_fullsized":"2024-03-29T04:08:54.815991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925275+0000","last_change":"2024-03-29T04:08:52.809190+0000","last_active":"2024-03-29T04:08:55.925275+0000","last_peered":"2024-03-29T04:08:55.925275+0000","last_clean":"2024-03-29T04:08:55.925275+0000","last_became_active":"2024-03-29T04:08:52.808923+0000","last_became_peered":"2024-03-29T04:08:52.808923+0000","last_unstale":"2024-03-29T04:08:55.925275+0000","last_undegraded":"2024-03-29T04:08:55.925275+0000","last_fullsized":"2024-03-29T04:08:55.925275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924506+0000","last_change":"2024-03-29T04:08:53.828865+0000","last_active":"2024-03-29T04:08:55.924506+0000","last_peered":"2024-03-29T04:08:55.924506+0000","last_clean":"2024-03-29T04:08:55.924506+0000","last_became_active":"2024-03-29T04:08:53.828601+0000","last_became_peered":"2024-03-29T04:08:53.828601+0000","last_unstale":"2024-03-29T04:08:55.924506+0000","last_undegraded":"2024-03-29T04:08:55.924506+0000","last_fullsized":"2024-03-29T04:08:55.924506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814227+0000","last_change":"2024-03-29T04:08:52.810336+0000","last_active":"2024-03-29T04:08:54.814227+0000","last_peered":"2024-03-29T04:08:54.814227+0000","last_clean":"2024-03-29T04:08:54.814227+0000","last_became_active":"2024-03-29T04:08:52.810021+0000","last_became_peered":"2024-03-29T04:08:52.810021+0000","last_unstale":"2024-03-29T04:08:54.814227+0000","last_undegraded":"2024-03-29T04:08:54.814227+0000","last_fullsized":"2024-03-29T04:08:54.814227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811960+0000","last_change":"2024-03-29T04:08:53.817702+0000","last_active":"2024-03-29T04:08:54.811960+0000","last_peered":"2024-03-29T04:08:54.811960+0000","last_clean":"2024-03-29T04:08:54.811960+0000","last_became_active":"2024-03-29T04:08:53.817117+0000","last_became_peered":"2024-03-29T04:08:53.817117+0000","last_unstale":"2024-03-29T04:08:54.811960+0000","last_undegraded":"2024-03-29T04:08:54.811960+0000","last_fullsized":"2024-03-29T04:08:54.811960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816011+0000","last_change":"2024-03-29T04:08:52.812898+0000","last_active":"2024-03-29T04:08:54.816011+0000","last_peered":"2024-03-29T04:08:54.816011+0000","last_clean":"2024-03-29T04:08:54.816011+0000","last_became_active":"2024-03-29T04:08:52.811143+0000","last_became_peered":"2024-03-29T04:08:52.811143+0000","last_unstale":"2024-03-29T04:08:54.816011+0000","last_undegraded":"2024-03-29T04:08:54.816011+0000","last_fullsized":"2024-03-29T04:08:54.816011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813759+0000","last_change":"2024-03-29T04:08:53.815139+0000","last_active":"2024-03-29T04:08:54.813759+0000","last_peered":"2024-03-29T04:08:54.813759+0000","last_clean":"2024-03-29T04:08:54.813759+0000","last_became_active":"2024-03-29T04:08:53.814836+0000","last_became_peered":"2024-03-29T04:08:53.814836+0000","last_unstale":"2024-03-29T04:08:54.813759+0000","last_undegraded":"2024-03-29T04:08:54.813759+0000","last_fullsized":"2024-03-29T04:08:54.813759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813822+0000","last_change":"2024-03-29T04:08:52.812383+0000","last_active":"2024-03-29T04:08:54.813822+0000","last_peered":"2024-03-29T04:08:54.813822+0000","last_clean":"2024-03-29T04:08:54.813822+0000","last_became_active":"2024-03-29T04:08:52.810343+0000","last_became_peered":"2024-03-29T04:08:52.810343+0000","last_unstale":"2024-03-29T04:08:54.813822+0000","last_undegraded":"2024-03-29T04:08:54.813822+0000","last_fullsized":"2024-03-29T04:08:54.813822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924573+0000","last_change":"2024-03-29T04:08:53.821898+0000","last_active":"2024-03-29T04:08:55.924573+0000","last_peered":"2024-03-29T04:08:55.924573+0000","last_clean":"2024-03-29T04:08:55.924573+0000","last_became_active":"2024-03-29T04:08:53.821126+0000","last_became_peered":"2024-03-29T04:08:53.821126+0000","last_unstale":"2024-03-29T04:08:55.924573+0000","last_undegraded":"2024-03-29T04:08:55.924573+0000","last_fullsized":"2024-03-29T04:08:55.924573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812079+0000","last_change":"2024-03-29T04:08:52.810835+0000","last_active":"2024-03-29T04:08:54.812079+0000","last_peered":"2024-03-29T04:08:54.812079+0000","last_clean":"2024-03-29T04:08:54.812079+0000","last_became_active":"2024-03-29T04:08:52.810585+0000","last_became_peered":"2024-03-29T04:08:52.810585+0000","last_unstale":"2024-03-29T04:08:54.812079+0000","last_undegraded":"2024-03-29T04:08:54.812079+0000","last_fullsized":"2024-03-29T04:08:54.812079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814427+0000","last_change":"2024-03-29T04:08:53.817718+0000","last_active":"2024-03-29T04:08:54.814427+0000","last_peered":"2024-03-29T04:08:54.814427+0000","last_clean":"2024-03-29T04:08:54.814427+0000","last_became_active":"2024-03-29T04:08:53.817419+0000","last_became_peered":"2024-03-29T04:08:53.817419+0000","last_unstale":"2024-03-29T04:08:54.814427+0000","last_undegraded":"2024-03-29T04:08:54.814427+0000","last_fullsized":"2024-03-29T04:08:54.814427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816294+0000","last_change":"2024-03-29T04:08:52.809428+0000","last_active":"2024-03-29T04:08:54.816294+0000","last_peered":"2024-03-29T04:08:54.816294+0000","last_clean":"2024-03-29T04:08:54.816294+0000","last_became_active":"2024-03-29T04:08:52.809124+0000","last_became_peered":"2024-03-29T04:08:52.809124+0000","last_unstale":"2024-03-29T04:08:54.816294+0000","last_undegraded":"2024-03-29T04:08:54.816294+0000","last_fullsized":"2024-03-29T04:08:54.816294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546068+0000","last_change":"2024-03-29T04:08:53.823456+0000","last_active":"2024-03-29T04:08:55.546068+0000","last_peered":"2024-03-29T04:08:55.546068+0000","last_clean":"2024-03-29T04:08:55.546068+0000","last_became_active":"2024-03-29T04:08:53.823196+0000","last_became_peered":"2024-03-29T04:08:53.823196+0000","last_unstale":"2024-03-29T04:08:55.546068+0000","last_undegraded":"2024-03-29T04:08:55.546068+0000","last_fullsized":"2024-03-29T04:08:55.546068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924849+0000","last_change":"2024-03-29T04:08:52.804947+0000","last_active":"2024-03-29T04:08:55.924849+0000","last_peered":"2024-03-29T04:08:55.924849+0000","last_clean":"2024-03-29T04:08:55.924849+0000","last_became_active":"2024-03-29T04:08:52.804675+0000","last_became_peered":"2024-03-29T04:08:52.804675+0000","last_unstale":"2024-03-29T04:08:55.924849+0000","last_undegraded":"2024-03-29T04:08:55.924849+0000","last_fullsized":"2024-03-29T04:08:55.924849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924675+0000","last_change":"2024-03-29T04:08:53.815297+0000","last_active":"2024-03-29T04:08:55.924675+0000","last_peered":"2024-03-29T04:08:55.924675+0000","last_clean":"2024-03-29T04:08:55.924675+0000","last_became_active":"2024-03-29T04:08:53.815003+0000","last_became_peered":"2024-03-29T04:08:53.815003+0000","last_unstale":"2024-03-29T04:08:55.924675+0000","last_undegraded":"2024-03-29T04:08:55.924675+0000","last_fullsized":"2024-03-29T04:08:55.924675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814596+0000","last_change":"2024-03-29T04:08:52.809942+0000","last_active":"2024-03-29T04:08:54.814596+0000","last_peered":"2024-03-29T04:08:54.814596+0000","last_clean":"2024-03-29T04:08:54.814596+0000","last_became_active":"2024-03-29T04:08:52.809671+0000","last_became_peered":"2024-03-29T04:08:52.809671+0000","last_unstale":"2024-03-29T04:08:54.814596+0000","last_undegraded":"2024-03-29T04:08:54.814596+0000","last_fullsized":"2024-03-29T04:08:54.814596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546150+0000","last_change":"2024-03-29T04:08:53.818585+0000","last_active":"2024-03-29T04:08:55.546150+0000","last_peered":"2024-03-29T04:08:55.546150+0000","last_clean":"2024-03-29T04:08:55.546150+0000","last_became_active":"2024-03-29T04:08:53.818252+0000","last_became_peered":"2024-03-29T04:08:53.818252+0000","last_unstale":"2024-03-29T04:08:55.546150+0000","last_undegraded":"2024-03-29T04:08:55.546150+0000","last_fullsized":"2024-03-29T04:08:55.546150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815950+0000","last_change":"2024-03-29T04:08:52.810780+0000","last_active":"2024-03-29T04:08:54.815950+0000","last_peered":"2024-03-29T04:08:54.815950+0000","last_clean":"2024-03-29T04:08:54.815950+0000","last_became_active":"2024-03-29T04:08:52.810487+0000","last_became_peered":"2024-03-29T04:08:52.810487+0000","last_unstale":"2024-03-29T04:08:54.815950+0000","last_undegraded":"2024-03-29T04:08:54.815950+0000","last_fullsized":"2024-03-29T04:08:54.815950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816134+0000","last_change":"2024-03-29T04:08:53.825665+0000","last_active":"2024-03-29T04:08:54.816134+0000","last_peered":"2024-03-29T04:08:54.816134+0000","last_clean":"2024-03-29T04:08:54.816134+0000","last_became_active":"2024-03-29T04:08:53.825229+0000","last_became_peered":"2024-03-29T04:08:53.825229+0000","last_unstale":"2024-03-29T04:08:54.816134+0000","last_undegraded":"2024-03-29T04:08:54.816134+0000","last_fullsized":"2024-03-29T04:08:54.816134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924756+0000","last_change":"2024-03-29T04:08:52.807491+0000","last_active":"2024-03-29T04:08:55.924756+0000","last_peered":"2024-03-29T04:08:55.924756+0000","last_clean":"2024-03-29T04:08:55.924756+0000","last_became_active":"2024-03-29T04:08:52.807236+0000","last_became_peered":"2024-03-29T04:08:52.807236+0000","last_unstale":"2024-03-29T04:08:55.924756+0000","last_undegraded":"2024-03-29T04:08:55.924756+0000","last_fullsized":"2024-03-29T04:08:55.924756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545849+0000","last_change":"2024-03-29T04:08:53.819664+0000","last_active":"2024-03-29T04:08:55.545849+0000","last_peered":"2024-03-29T04:08:55.545849+0000","last_clean":"2024-03-29T04:08:55.545849+0000","last_became_active":"2024-03-29T04:08:53.819414+0000","last_became_peered":"2024-03-29T04:08:53.819414+0000","last_unstale":"2024-03-29T04:08:55.545849+0000","last_undegraded":"2024-03-29T04:08:55.545849+0000","last_fullsized":"2024-03-29T04:08:55.545849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815063+0000","last_change":"2024-03-29T04:08:52.806445+0000","last_active":"2024-03-29T04:08:54.815063+0000","last_peered":"2024-03-29T04:08:54.815063+0000","last_clean":"2024-03-29T04:08:54.815063+0000","last_became_active":"2024-03-29T04:08:52.806166+0000","last_became_peered":"2024-03-29T04:08:52.806166+0000","last_unstale":"2024-03-29T04:08:54.815063+0000","last_undegraded":"2024-03-29T04:08:54.815063+0000","last_fullsized":"2024-03-29T04:08:54.815063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:03.546395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546168+0000","last_change":"2024-03-29T04:08:53.819783+0000","last_active":"2024-03-29T04:08:55.546168+0000","last_peered":"2024-03-29T04:08:55.546168+0000","last_clean":"2024-03-29T04:08:55.546168+0000","last_became_active":"2024-03-29T04:08:53.819507+0000","last_became_peered":"2024-03-29T04:08:53.819507+0000","last_unstale":"2024-03-29T04:08:55.546168+0000","last_undegraded":"2024-03-29T04:08:55.546168+0000","last_fullsized":"2024-03-29T04:08:55.546168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924833+0000","last_change":"2024-03-29T04:08:52.812349+0000","last_active":"2024-03-29T04:08:55.924833+0000","last_peered":"2024-03-29T04:08:55.924833+0000","last_clean":"2024-03-29T04:08:55.924833+0000","last_became_active":"2024-03-29T04:08:52.812081+0000","last_became_peered":"2024-03-29T04:08:52.812081+0000","last_unstale":"2024-03-29T04:08:55.924833+0000","last_undegraded":"2024-03-29T04:08:55.924833+0000","last_fullsized":"2024-03-29T04:08:55.924833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546387+0000","last_change":"2024-03-29T04:08:53.827351+0000","last_active":"2024-03-29T04:08:55.546387+0000","last_peered":"2024-03-29T04:08:55.546387+0000","last_clean":"2024-03-29T04:08:55.546387+0000","last_became_active":"2024-03-29T04:08:53.826255+0000","last_became_peered":"2024-03-29T04:08:53.826255+0000","last_unstale":"2024-03-29T04:08:55.546387+0000","last_undegraded":"2024-03-29T04:08:55.546387+0000","last_fullsized":"2024-03-29T04:08:55.546387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812164+0000","last_change":"2024-03-29T04:08:52.805752+0000","last_active":"2024-03-29T04:08:54.812164+0000","last_peered":"2024-03-29T04:08:54.812164+0000","last_clean":"2024-03-29T04:08:54.812164+0000","last_became_active":"2024-03-29T04:08:52.804921+0000","last_became_peered":"2024-03-29T04:08:52.804921+0000","last_unstale":"2024-03-29T04:08:54.812164+0000","last_undegraded":"2024-03-29T04:08:54.812164+0000","last_fullsized":"2024-03-29T04:08:54.812164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815243+0000","last_change":"2024-03-29T04:08:53.821399+0000","last_active":"2024-03-29T04:08:54.815243+0000","last_peered":"2024-03-29T04:08:54.815243+0000","last_clean":"2024-03-29T04:08:54.815243+0000","last_became_active":"2024-03-29T04:08:53.821026+0000","last_became_peered":"2024-03-29T04:08:53.821026+0000","last_unstale":"2024-03-29T04:08:54.815243+0000","last_undegraded":"2024-03-29T04:08:54.815243+0000","last_fullsized":"2024-03-29T04:08:54.815243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924593+0000","last_change":"2024-03-29T04:08:52.811397+0000","last_active":"2024-03-29T04:08:55.924593+0000","last_peered":"2024-03-29T04:08:55.924593+0000","last_clean":"2024-03-29T04:08:55.924593+0000","last_became_active":"2024-03-29T04:08:52.810968+0000","last_became_peered":"2024-03-29T04:08:52.810968+0000","last_unstale":"2024-03-29T04:08:55.924593+0000","last_undegraded":"2024-03-29T04:08:55.924593+0000","last_fullsized":"2024-03-29T04:08:55.924593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546224+0000","last_change":"2024-03-29T04:08:53.824444+0000","last_active":"2024-03-29T04:08:55.546224+0000","last_peered":"2024-03-29T04:08:55.546224+0000","last_clean":"2024-03-29T04:08:55.546224+0000","last_became_active":"2024-03-29T04:08:53.821578+0000","last_became_peered":"2024-03-29T04:08:53.821578+0000","last_unstale":"2024-03-29T04:08:55.546224+0000","last_undegraded":"2024-03-29T04:08:55.546224+0000","last_fullsized":"2024-03-29T04:08:55.546224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545812+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:08:55.545812+0000","last_peered":"2024-03-29T04:08:55.545812+0000","last_clean":"2024-03-29T04:08:55.545812+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:08:55.545812+0000","last_undegraded":"2024-03-29T04:08:55.545812+0000","last_fullsized":"2024-03-29T04:08:55.545812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813649+0000","last_change":"2024-03-29T04:08:52.807738+0000","last_active":"2024-03-29T04:08:54.813649+0000","last_peered":"2024-03-29T04:08:54.813649+0000","last_clean":"2024-03-29T04:08:54.813649+0000","last_became_active":"2024-03-29T04:08:52.807442+0000","last_became_peered":"2024-03-29T04:08:52.807442+0000","last_unstale":"2024-03-29T04:08:54.813649+0000","last_undegraded":"2024-03-29T04:08:54.813649+0000","last_fullsized":"2024-03-29T04:08:54.813649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925459+0000","last_change":"2024-03-29T04:08:53.825630+0000","last_active":"2024-03-29T04:08:55.925459+0000","last_peered":"2024-03-29T04:08:55.925459+0000","last_clean":"2024-03-29T04:08:55.925459+0000","last_became_active":"2024-03-29T04:08:53.825386+0000","last_became_peered":"2024-03-29T04:08:53.825386+0000","last_unstale":"2024-03-29T04:08:55.925459+0000","last_undegraded":"2024-03-29T04:08:55.925459+0000","last_fullsized":"2024-03-29T04:08:55.925459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545708+0000","last_change":"2024-03-29T04:08:52.820187+0000","last_active":"2024-03-29T04:08:55.545708+0000","last_peered":"2024-03-29T04:08:55.545708+0000","last_clean":"2024-03-29T04:08:55.545708+0000","last_became_active":"2024-03-29T04:08:52.819912+0000","last_became_peered":"2024-03-29T04:08:52.819912+0000","last_unstale":"2024-03-29T04:08:55.545708+0000","last_undegraded":"2024-03-29T04:08:55.545708+0000","last_fullsized":"2024-03-29T04:08:55.545708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924298+0000","last_change":"2024-03-29T04:08:53.815607+0000","last_active":"2024-03-29T04:08:55.924298+0000","last_peered":"2024-03-29T04:08:55.924298+0000","last_clean":"2024-03-29T04:08:55.924298+0000","last_became_active":"2024-03-29T04:08:53.814274+0000","last_became_peered":"2024-03-29T04:08:53.814274+0000","last_unstale":"2024-03-29T04:08:55.924298+0000","last_undegraded":"2024-03-29T04:08:55.924298+0000","last_fullsized":"2024-03-29T04:08:55.924298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:39.590764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924274+0000","last_change":"2024-03-29T04:08:52.813143+0000","last_active":"2024-03-29T04:08:55.924274+0000","last_peered":"2024-03-29T04:08:55.924274+0000","last_clean":"2024-03-29T04:08:55.924274+0000","last_became_active":"2024-03-29T04:08:52.812866+0000","last_became_peered":"2024-03-29T04:08:52.812866+0000","last_unstale":"2024-03-29T04:08:55.924274+0000","last_undegraded":"2024-03-29T04:08:55.924274+0000","last_fullsized":"2024-03-29T04:08:55.924274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811826+0000","last_change":"2024-03-29T04:08:53.818488+0000","last_active":"2024-03-29T04:08:54.811826+0000","last_peered":"2024-03-29T04:08:54.811826+0000","last_clean":"2024-03-29T04:08:54.811826+0000","last_became_active":"2024-03-29T04:08:53.818222+0000","last_became_peered":"2024-03-29T04:08:53.818222+0000","last_unstale":"2024-03-29T04:08:54.811826+0000","last_undegraded":"2024-03-29T04:08:54.811826+0000","last_fullsized":"2024-03-29T04:08:54.811826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814714+0000","last_change":"2024-03-29T04:08:52.808830+0000","last_active":"2024-03-29T04:08:54.814714+0000","last_peered":"2024-03-29T04:08:54.814714+0000","last_clean":"2024-03-29T04:08:54.814714+0000","last_became_active":"2024-03-29T04:08:52.808524+0000","last_became_peered":"2024-03-29T04:08:52.808524+0000","last_unstale":"2024-03-29T04:08:54.814714+0000","last_undegraded":"2024-03-29T04:08:54.814714+0000","last_fullsized":"2024-03-29T04:08:54.814714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815259+0000","last_change":"2024-03-29T04:08:53.823062+0000","last_active":"2024-03-29T04:08:54.815259+0000","last_peered":"2024-03-29T04:08:54.815259+0000","last_clean":"2024-03-29T04:08:54.815259+0000","last_became_active":"2024-03-29T04:08:53.822666+0000","last_became_peered":"2024-03-29T04:08:53.822666+0000","last_unstale":"2024-03-29T04:08:54.815259+0000","last_undegraded":"2024-03-29T04:08:54.815259+0000","last_fullsized":"2024-03-29T04:08:54.815259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812019+0000","last_change":"2024-03-29T04:08:52.799226+0000","last_active":"2024-03-29T04:08:54.812019+0000","last_peered":"2024-03-29T04:08:54.812019+0000","last_clean":"2024-03-29T04:08:54.812019+0000","last_became_active":"2024-03-29T04:08:52.798964+0000","last_became_peered":"2024-03-29T04:08:52.798964+0000","last_unstale":"2024-03-29T04:08:54.812019+0000","last_undegraded":"2024-03-29T04:08:54.812019+0000","last_fullsized":"2024-03-29T04:08:54.812019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815451+0000","last_change":"2024-03-29T04:08:53.821737+0000","last_active":"2024-03-29T04:08:54.815451+0000","last_peered":"2024-03-29T04:08:54.815451+0000","last_clean":"2024-03-29T04:08:54.815451+0000","last_became_active":"2024-03-29T04:08:53.820680+0000","last_became_peered":"2024-03-29T04:08:53.820680+0000","last_unstale":"2024-03-29T04:08:54.815451+0000","last_undegraded":"2024-03-29T04:08:54.815451+0000","last_fullsized":"2024-03-29T04:08:54.815451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546333+0000","last_change":"2024-03-29T04:08:52.821118+0000","last_active":"2024-03-29T04:08:55.546333+0000","last_peered":"2024-03-29T04:08:55.546333+0000","last_clean":"2024-03-29T04:08:55.546333+0000","last_became_active":"2024-03-29T04:08:52.820549+0000","last_became_peered":"2024-03-29T04:08:52.820549+0000","last_unstale":"2024-03-29T04:08:55.546333+0000","last_undegraded":"2024-03-29T04:08:55.546333+0000","last_fullsized":"2024-03-29T04:08:55.546333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811978+0000","last_change":"2024-03-29T04:08:53.810781+0000","last_active":"2024-03-29T04:08:54.811978+0000","last_peered":"2024-03-29T04:08:54.811978+0000","last_clean":"2024-03-29T04:08:54.811978+0000","last_became_active":"2024-03-29T04:08:53.810518+0000","last_became_peered":"2024-03-29T04:08:53.810518+0000","last_unstale":"2024-03-29T04:08:54.811978+0000","last_undegraded":"2024-03-29T04:08:54.811978+0000","last_fullsized":"2024-03-29T04:08:54.811978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816447+0000","last_change":"2024-03-29T04:08:52.809876+0000","last_active":"2024-03-29T04:08:54.816447+0000","last_peered":"2024-03-29T04:08:54.816447+0000","last_clean":"2024-03-29T04:08:54.816447+0000","last_became_active":"2024-03-29T04:08:52.809575+0000","last_became_peered":"2024-03-29T04:08:52.809575+0000","last_unstale":"2024-03-29T04:08:54.816447+0000","last_undegraded":"2024-03-29T04:08:54.816447+0000","last_fullsized":"2024-03-29T04:08:54.816447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925771+0000","last_change":"2024-03-29T04:08:53.831225+0000","last_active":"2024-03-29T04:08:55.925771+0000","last_peered":"2024-03-29T04:08:55.925771+0000","last_clean":"2024-03-29T04:08:55.925771+0000","last_became_active":"2024-03-29T04:08:53.830819+0000","last_became_peered":"2024-03-29T04:08:53.830819+0000","last_unstale":"2024-03-29T04:08:55.925771+0000","last_undegraded":"2024-03-29T04:08:55.925771+0000","last_fullsized":"2024-03-29T04:08:55.925771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816132+0000","last_change":"2024-03-29T04:08:52.812725+0000","last_active":"2024-03-29T04:08:54.816132+0000","last_peered":"2024-03-29T04:08:54.816132+0000","last_clean":"2024-03-29T04:08:54.816132+0000","last_became_active":"2024-03-29T04:08:52.812020+0000","last_became_peered":"2024-03-29T04:08:52.812020+0000","last_unstale":"2024-03-29T04:08:54.816132+0000","last_undegraded":"2024-03-29T04:08:54.816132+0000","last_fullsized":"2024-03-29T04:08:54.816132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925009+0000","last_change":"2024-03-29T04:08:53.811809+0000","last_active":"2024-03-29T04:08:55.925009+0000","last_peered":"2024-03-29T04:08:55.925009+0000","last_clean":"2024-03-29T04:08:55.925009+0000","last_became_active":"2024-03-29T04:08:53.811513+0000","last_became_peered":"2024-03-29T04:08:53.811513+0000","last_unstale":"2024-03-29T04:08:55.925009+0000","last_undegraded":"2024-03-29T04:08:55.925009+0000","last_fullsized":"2024-03-29T04:08:55.925009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545699+0000","last_change":"2024-03-29T04:08:52.814972+0000","last_active":"2024-03-29T04:08:55.545699+0000","last_peered":"2024-03-29T04:08:55.545699+0000","last_clean":"2024-03-29T04:08:55.545699+0000","last_became_active":"2024-03-29T04:08:52.814714+0000","last_became_peered":"2024-03-29T04:08:52.814714+0000","last_unstale":"2024-03-29T04:08:55.545699+0000","last_undegraded":"2024-03-29T04:08:55.545699+0000","last_fullsized":"2024-03-29T04:08:55.545699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814365+0000","last_change":"2024-03-29T04:08:53.817677+0000","last_active":"2024-03-29T04:08:54.814365+0000","last_peered":"2024-03-29T04:08:54.814365+0000","last_clean":"2024-03-29T04:08:54.814365+0000","last_became_active":"2024-03-29T04:08:53.815481+0000","last_became_peered":"2024-03-29T04:08:53.815481+0000","last_unstale":"2024-03-29T04:08:54.814365+0000","last_undegraded":"2024-03-29T04:08:54.814365+0000","last_fullsized":"2024-03-29T04:08:54.814365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811927+0000","last_change":"2024-03-29T04:08:52.814031+0000","last_active":"2024-03-29T04:08:54.811927+0000","last_peered":"2024-03-29T04:08:54.811927+0000","last_clean":"2024-03-29T04:08:54.811927+0000","last_became_active":"2024-03-29T04:08:52.813714+0000","last_became_peered":"2024-03-29T04:08:52.813714+0000","last_unstale":"2024-03-29T04:08:54.811927+0000","last_undegraded":"2024-03-29T04:08:54.811927+0000","last_fullsized":"2024-03-29T04:08:54.811927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924641+0000","last_change":"2024-03-29T04:08:53.818653+0000","last_active":"2024-03-29T04:08:55.924641+0000","last_peered":"2024-03-29T04:08:55.924641+0000","last_clean":"2024-03-29T04:08:55.924641+0000","last_became_active":"2024-03-29T04:08:53.818387+0000","last_became_peered":"2024-03-29T04:08:53.818387+0000","last_unstale":"2024-03-29T04:08:55.924641+0000","last_undegraded":"2024-03-29T04:08:55.924641+0000","last_fullsized":"2024-03-29T04:08:55.924641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812062+0000","last_change":"2024-03-29T04:08:52.806456+0000","last_active":"2024-03-29T04:08:54.812062+0000","last_peered":"2024-03-29T04:08:54.812062+0000","last_clean":"2024-03-29T04:08:54.812062+0000","last_became_active":"2024-03-29T04:08:52.806184+0000","last_became_peered":"2024-03-29T04:08:52.806184+0000","last_unstale":"2024-03-29T04:08:54.812062+0000","last_undegraded":"2024-03-29T04:08:54.812062+0000","last_fullsized":"2024-03-29T04:08:54.812062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925468+0000","last_change":"2024-03-29T04:08:53.828867+0000","last_active":"2024-03-29T04:08:55.925468+0000","last_peered":"2024-03-29T04:08:55.925468+0000","last_clean":"2024-03-29T04:08:55.925468+0000","last_became_active":"2024-03-29T04:08:53.828610+0000","last_became_peered":"2024-03-29T04:08:53.828610+0000","last_unstale":"2024-03-29T04:08:55.925468+0000","last_undegraded":"2024-03-29T04:08:55.925468+0000","last_fullsized":"2024-03-29T04:08:55.925468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546226+0000","last_change":"2024-03-29T04:08:52.821021+0000","last_active":"2024-03-29T04:08:55.546226+0000","last_peered":"2024-03-29T04:08:55.546226+0000","last_clean":"2024-03-29T04:08:55.546226+0000","last_became_active":"2024-03-29T04:08:52.820478+0000","last_became_peered":"2024-03-29T04:08:52.820478+0000","last_unstale":"2024-03-29T04:08:55.546226+0000","last_undegraded":"2024-03-29T04:08:55.546226+0000","last_fullsized":"2024-03-29T04:08:55.546226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815868+0000","last_change":"2024-03-29T04:08:53.825032+0000","last_active":"2024-03-29T04:08:54.815868+0000","last_peered":"2024-03-29T04:08:54.815868+0000","last_clean":"2024-03-29T04:08:54.815868+0000","last_became_active":"2024-03-29T04:08:53.824740+0000","last_became_peered":"2024-03-29T04:08:53.824740+0000","last_unstale":"2024-03-29T04:08:54.815868+0000","last_undegraded":"2024-03-29T04:08:54.815868+0000","last_fullsized":"2024-03-29T04:08:54.815868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815425+0000","last_change":"2024-03-29T04:08:53.816669+0000","last_active":"2024-03-29T04:08:54.815425+0000","last_peered":"2024-03-29T04:08:54.815425+0000","last_clean":"2024-03-29T04:08:54.815425+0000","last_became_active":"2024-03-29T04:08:53.816348+0000","last_became_peered":"2024-03-29T04:08:53.816348+0000","last_unstale":"2024-03-29T04:08:54.815425+0000","last_undegraded":"2024-03-29T04:08:54.815425+0000","last_fullsized":"2024-03-29T04:08:54.815425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545977+0000","last_change":"2024-03-29T04:08:52.820909+0000","last_active":"2024-03-29T04:08:55.545977+0000","last_peered":"2024-03-29T04:08:55.545977+0000","last_clean":"2024-03-29T04:08:55.545977+0000","last_became_active":"2024-03-29T04:08:52.820523+0000","last_became_peered":"2024-03-29T04:08:52.820523+0000","last_unstale":"2024-03-29T04:08:55.545977+0000","last_undegraded":"2024-03-29T04:08:55.545977+0000","last_fullsized":"2024-03-29T04:08:55.545977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815127+0000","last_change":"2024-03-29T04:08:53.817769+0000","last_active":"2024-03-29T04:08:54.815127+0000","last_peered":"2024-03-29T04:08:54.815127+0000","last_clean":"2024-03-29T04:08:54.815127+0000","last_became_active":"2024-03-29T04:08:53.816753+0000","last_became_peered":"2024-03-29T04:08:53.816753+0000","last_unstale":"2024-03-29T04:08:54.815127+0000","last_undegraded":"2024-03-29T04:08:54.815127+0000","last_fullsized":"2024-03-29T04:08:54.815127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546206+0000","last_change":"2024-03-29T04:08:52.820167+0000","last_active":"2024-03-29T04:08:55.546206+0000","last_peered":"2024-03-29T04:08:55.546206+0000","last_clean":"2024-03-29T04:08:55.546206+0000","last_became_active":"2024-03-29T04:08:52.819858+0000","last_became_peered":"2024-03-29T04:08:52.819858+0000","last_unstale":"2024-03-29T04:08:55.546206+0000","last_undegraded":"2024-03-29T04:08:55.546206+0000","last_fullsized":"2024-03-29T04:08:55.546206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924413+0000","last_change":"2024-03-29T04:08:53.822001+0000","last_active":"2024-03-29T04:08:55.924413+0000","last_peered":"2024-03-29T04:08:55.924413+0000","last_clean":"2024-03-29T04:08:55.924413+0000","last_became_active":"2024-03-29T04:08:53.821669+0000","last_became_peered":"2024-03-29T04:08:53.821669+0000","last_unstale":"2024-03-29T04:08:55.924413+0000","last_undegraded":"2024-03-29T04:08:55.924413+0000","last_fullsized":"2024-03-29T04:08:55.924413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925665+0000","last_change":"2024-03-29T04:08:52.807347+0000","last_active":"2024-03-29T04:08:55.925665+0000","last_peered":"2024-03-29T04:08:55.925665+0000","last_clean":"2024-03-29T04:08:55.925665+0000","last_became_active":"2024-03-29T04:08:52.807042+0000","last_became_peered":"2024-03-29T04:08:52.807042+0000","last_unstale":"2024-03-29T04:08:55.925665+0000","last_undegraded":"2024-03-29T04:08:55.925665+0000","last_fullsized":"2024-03-29T04:08:55.925665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925593+0000","last_change":"2024-03-29T04:08:53.829560+0000","last_active":"2024-03-29T04:08:55.925593+0000","last_peered":"2024-03-29T04:08:55.925593+0000","last_clean":"2024-03-29T04:08:55.925593+0000","last_became_active":"2024-03-29T04:08:53.829312+0000","last_became_peered":"2024-03-29T04:08:53.829312+0000","last_unstale":"2024-03-29T04:08:55.925593+0000","last_undegraded":"2024-03-29T04:08:55.925593+0000","last_fullsized":"2024-03-29T04:08:55.925593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924658+0000","last_change":"2024-03-29T04:08:52.813322+0000","last_active":"2024-03-29T04:08:55.924658+0000","last_peered":"2024-03-29T04:08:55.924658+0000","last_clean":"2024-03-29T04:08:55.924658+0000","last_became_active":"2024-03-29T04:08:52.813020+0000","last_became_peered":"2024-03-29T04:08:52.813020+0000","last_unstale":"2024-03-29T04:08:55.924658+0000","last_undegraded":"2024-03-29T04:08:55.924658+0000","last_fullsized":"2024-03-29T04:08:55.924658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":665435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":225280,"data_stored":76097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":74485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":74645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":819200,"data_stored":665491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:09:03.467 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:09:03.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.625+0000 7fe1dfaf2640 1 Processor -- start 2024-03-29T04:09:03.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.625+0000 7fe1dfaf2640 1 -- start start 2024-03-29T04:09:03.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dfaf2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d815fba0 0x7fe1d8160a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dfaf2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe1d80b1850 0x7fe1d80b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dfaf2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d8160fa0 con 0x7fe1d80b32e0 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dfaf2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe1d80b1d00 con 0x7fe1d80b1850 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dfaf2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d80b1fe0 con 0x7fe1d815fba0 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe1d80b1850 0x7fe1d80b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe1d80b1850 0x7fe1d80b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48532/0 (socket says 172.21.15.156:48532) 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 -- 172.21.15.156:0/3793380808 learned_addr learned my addr 172.21.15.156:0/3793380808 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd88e640 1 --2- 172.21.15.156:0/3793380808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d815fba0 0x7fe1d8160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 -- 172.21.15.156:0/3793380808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d815fba0 msgr2=0x7fe1d8160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 --2- 172.21.15.156:0/3793380808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d815fba0 0x7fe1d8160a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 -- 172.21.15.156:0/3793380808 >> v1:172.21.15.203:6789/0 conn(0x7fe1d80b32e0 legacy=0x7fe1d809aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.626+0000 7fe1dd08d640 1 -- 172.21.15.156:0/3793380808 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d80b22c0 con 0x7fe1d80b1850 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.627+0000 7fe1dd08d640 1 --2- 172.21.15.156:0/3793380808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b1850 0x7fe1d80b2da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009870 tx=0x7fe1c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469ac9c51dff70f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.627+0000 7fe1dc88c640 1 -- 172.21.15.156:0/3793380808 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8035020 con 0x7fe1d80b1850 2024-03-29T04:09:03.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.627+0000 7fe1dc88c640 1 -- 172.21.15.156:0/3793380808 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c8031640 con 0x7fe1d80b1850 2024-03-29T04:09:03.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.627+0000 7fe1dc88c640 1 -- 172.21.15.156:0/3793380808 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c802d040 con 0x7fe1d80b1850 2024-03-29T04:09:03.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.627+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/3793380808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b1850 msgr2=0x7fe1d80b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.627+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/3793380808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b1850 0x7fe1d80b2da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009870 tx=0x7fe1c802b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.628+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/3793380808 shutdown_connections 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.628+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/3793380808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b1850 0x7fe1d80b2da0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.628+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/3793380808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d815fba0 0x7fe1d8160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.628+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/3793380808 >> 172.21.15.156:0/3793380808 conn(0x7fe1d80b3d90 msgr2=0x7fe1d80b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.628+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/3793380808 shutdown_connections 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.628+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/3793380808 wait complete. 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.629+0000 7fe1dfaf2640 1 Processor -- start 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.629+0000 7fe1dfaf2640 1 -- start start 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.629+0000 7fe1dfaf2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d80b1850 0x7fe1d815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dfaf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 0x7fe1d80cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dfaf2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1d815fba0 0x7fe1d80cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dfaf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d80b3990 con 0x7fe1d80b32e0 2024-03-29T04:09:03.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dfaf2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d80b2880 con 0x7fe1d815fba0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dfaf2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d809b150 con 0x7fe1d80b1850 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 0x7fe1d80cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 0x7fe1d80cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48534/0 (socket says 172.21.15.156:48534) 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 -- 172.21.15.156:0/795598700 learned_addr learned my addr 172.21.15.156:0/795598700 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd88e640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d80b1850 0x7fe1d815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1de08f640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1d815fba0 0x7fe1d80cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 -- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d80b1850 msgr2=0x7fe1d815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d80b1850 0x7fe1d815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 -- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1d815fba0 msgr2=0x7fe1d80cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1d815fba0 0x7fe1d80cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 -- 172.21.15.156:0/795598700 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d80c6f90 con 0x7fe1d80b32e0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1dd08d640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 0x7fe1d80cba90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8000c00 tx=0x7fe1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.630+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8040070 con 0x7fe1d80b32e0 2024-03-29T04:09:03.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.631+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c80399f0 con 0x7fe1d80b32e0 2024-03-29T04:09:03.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.631+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d80c7270 con 0x7fe1d80b32e0 2024-03-29T04:09:03.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.631+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8035040 con 0x7fe1d80b32e0 2024-03-29T04:09:03.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.631+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d80c7830 con 0x7fe1d80b32e0 2024-03-29T04:09:03.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.632+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d8068d20 con 0x7fe1d80b32e0 2024-03-29T04:09:03.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.632+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe1c804b060 con 0x7fe1d80b32e0 2024-03-29T04:09:03.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.632+0000 7fe1ce7fc640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe1bc06c530 0x7fe1bc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.633+0000 7fe1dd88e640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe1bc06c530 0x7fe1bc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.633+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe1c80acf00 con 0x7fe1d80b32e0 2024-03-29T04:09:03.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.633+0000 7fe1dd88e640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe1bc06c530 0x7fe1bc06e9f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80c87d0 tx=0x7fe1d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.637+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe1c807b5b0 con 0x7fe1d80b32e0 2024-03-29T04:09:03.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.810+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1d8073bf0 con 0x7fe1bc06c530 2024-03-29T04:09:03.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.816+0000 7fe1ce7fc640 1 -- 172.21.15.156:0/795598700 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7fe1d8073bf0 con 0x7fe1bc06c530 2024-03-29T04:09:03.798 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:03.800 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:09:03.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe1bc06c530 msgr2=0x7fe1bc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe1bc06c530 0x7fe1bc06e9f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80c87d0 tx=0x7fe1d4008040 comp rx=0 tx=0).stop 2024-03-29T04:09:03.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 msgr2=0x7fe1d80cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 0x7fe1d80cba90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8000c00 tx=0x7fe1c80046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 shutdown_connections 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe1bc06c530 0x7fe1bc06e9f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1d815fba0 0x7fe1d80cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1d80b32e0 0x7fe1d80cba90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 --2- 172.21.15.156:0/795598700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1d80b1850 0x7fe1d815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 >> 172.21.15.156:0/795598700 conn(0x7fe1d80b3d90 msgr2=0x7fe1d8070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 shutdown_connections 2024-03-29T04:09:03.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.820+0000 7fe1dfaf2640 1 -- 172.21.15.156:0/795598700 wait complete. 2024-03-29T04:09:03.810 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T04:09:03.639316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235692,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739028,"statfs":{"total":773094113280,"available":772852764672,"internally_reserved":0,"allocated":2957312,"data_stored":1781436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.836674"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546238+0000","last_change":"2024-03-29T04:08:53.827540+0000","last_active":"2024-03-29T04:08:55.546238+0000","last_peered":"2024-03-29T04:08:55.546238+0000","last_clean":"2024-03-29T04:08:55.546238+0000","last_became_active":"2024-03-29T04:08:53.827301+0000","last_became_peered":"2024-03-29T04:08:53.827301+0000","last_unstale":"2024-03-29T04:08:55.546238+0000","last_undegraded":"2024-03-29T04:08:55.546238+0000","last_fullsized":"2024-03-29T04:08:55.546238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812222+0000","last_change":"2024-03-29T04:08:53.809513+0000","last_active":"2024-03-29T04:08:54.812222+0000","last_peered":"2024-03-29T04:08:54.812222+0000","last_clean":"2024-03-29T04:08:54.812222+0000","last_became_active":"2024-03-29T04:08:53.809236+0000","last_became_peered":"2024-03-29T04:08:53.809236+0000","last_unstale":"2024-03-29T04:08:54.812222+0000","last_undegraded":"2024-03-29T04:08:54.812222+0000","last_fullsized":"2024-03-29T04:08:54.812222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924239+0000","last_change":"2024-03-29T04:08:53.821233+0000","last_active":"2024-03-29T04:08:55.924239+0000","last_peered":"2024-03-29T04:08:55.924239+0000","last_clean":"2024-03-29T04:08:55.924239+0000","last_became_active":"2024-03-29T04:08:53.820557+0000","last_became_peered":"2024-03-29T04:08:53.820557+0000","last_unstale":"2024-03-29T04:08:55.924239+0000","last_undegraded":"2024-03-29T04:08:55.924239+0000","last_fullsized":"2024-03-29T04:08:55.924239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814954+0000","last_change":"2024-03-29T04:08:53.824851+0000","last_active":"2024-03-29T04:08:54.814954+0000","last_peered":"2024-03-29T04:08:54.814954+0000","last_clean":"2024-03-29T04:08:54.814954+0000","last_became_active":"2024-03-29T04:08:53.824563+0000","last_became_peered":"2024-03-29T04:08:53.824563+0000","last_unstale":"2024-03-29T04:08:54.814954+0000","last_undegraded":"2024-03-29T04:08:54.814954+0000","last_fullsized":"2024-03-29T04:08:54.814954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546094+0000","last_change":"2024-03-29T04:08:53.820567+0000","last_active":"2024-03-29T04:08:55.546094+0000","last_peered":"2024-03-29T04:08:55.546094+0000","last_clean":"2024-03-29T04:08:55.546094+0000","last_became_active":"2024-03-29T04:08:53.820290+0000","last_became_peered":"2024-03-29T04:08:53.820290+0000","last_unstale":"2024-03-29T04:08:55.546094+0000","last_undegraded":"2024-03-29T04:08:55.546094+0000","last_fullsized":"2024-03-29T04:08:55.546094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814434+0000","last_change":"2024-03-29T04:08:53.818966+0000","last_active":"2024-03-29T04:08:54.814434+0000","last_peered":"2024-03-29T04:08:54.814434+0000","last_clean":"2024-03-29T04:08:54.814434+0000","last_became_active":"2024-03-29T04:08:53.818396+0000","last_became_peered":"2024-03-29T04:08:53.818396+0000","last_unstale":"2024-03-29T04:08:54.814434+0000","last_undegraded":"2024-03-29T04:08:54.814434+0000","last_fullsized":"2024-03-29T04:08:54.814434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545835+0000","last_change":"2024-03-29T04:08:53.817600+0000","last_active":"2024-03-29T04:08:55.545835+0000","last_peered":"2024-03-29T04:08:55.545835+0000","last_clean":"2024-03-29T04:08:55.545835+0000","last_became_active":"2024-03-29T04:08:53.817308+0000","last_became_peered":"2024-03-29T04:08:53.817308+0000","last_unstale":"2024-03-29T04:08:55.545835+0000","last_undegraded":"2024-03-29T04:08:55.545835+0000","last_fullsized":"2024-03-29T04:08:55.545835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924368+0000","last_change":"2024-03-29T04:08:53.826247+0000","last_active":"2024-03-29T04:08:55.924368+0000","last_peered":"2024-03-29T04:08:55.924368+0000","last_clean":"2024-03-29T04:08:55.924368+0000","last_became_active":"2024-03-29T04:08:53.825974+0000","last_became_peered":"2024-03-29T04:08:53.825974+0000","last_unstale":"2024-03-29T04:08:55.924368+0000","last_undegraded":"2024-03-29T04:08:55.924368+0000","last_fullsized":"2024-03-29T04:08:55.924368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546330+0000","last_change":"2024-03-29T04:08:53.824041+0000","last_active":"2024-03-29T04:08:55.546330+0000","last_peered":"2024-03-29T04:08:55.546330+0000","last_clean":"2024-03-29T04:08:55.546330+0000","last_became_active":"2024-03-29T04:08:53.823788+0000","last_became_peered":"2024-03-29T04:08:53.823788+0000","last_unstale":"2024-03-29T04:08:55.546330+0000","last_undegraded":"2024-03-29T04:08:55.546330+0000","last_fullsized":"2024-03-29T04:08:55.546330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816424+0000","last_change":"2024-03-29T04:08:53.830205+0000","last_active":"2024-03-29T04:08:54.816424+0000","last_peered":"2024-03-29T04:08:54.816424+0000","last_clean":"2024-03-29T04:08:54.816424+0000","last_became_active":"2024-03-29T04:08:53.829931+0000","last_became_peered":"2024-03-29T04:08:53.829931+0000","last_unstale":"2024-03-29T04:08:54.816424+0000","last_undegraded":"2024-03-29T04:08:54.816424+0000","last_fullsized":"2024-03-29T04:08:54.816424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816205+0000","last_change":"2024-03-29T04:08:53.821458+0000","last_active":"2024-03-29T04:08:54.816205+0000","last_peered":"2024-03-29T04:08:54.816205+0000","last_clean":"2024-03-29T04:08:54.816205+0000","last_became_active":"2024-03-29T04:08:53.821168+0000","last_became_peered":"2024-03-29T04:08:53.821168+0000","last_unstale":"2024-03-29T04:08:54.816205+0000","last_undegraded":"2024-03-29T04:08:54.816205+0000","last_fullsized":"2024-03-29T04:08:54.816205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814279+0000","last_change":"2024-03-29T04:08:53.827058+0000","last_active":"2024-03-29T04:08:54.814279+0000","last_peered":"2024-03-29T04:08:54.814279+0000","last_clean":"2024-03-29T04:08:54.814279+0000","last_became_active":"2024-03-29T04:08:53.826167+0000","last_became_peered":"2024-03-29T04:08:53.826167+0000","last_unstale":"2024-03-29T04:08:54.814279+0000","last_undegraded":"2024-03-29T04:08:54.814279+0000","last_fullsized":"2024-03-29T04:08:54.814279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:22.521340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925452+0000","last_change":"2024-03-29T04:08:53.830239+0000","last_active":"2024-03-29T04:08:55.925452+0000","last_peered":"2024-03-29T04:08:55.925452+0000","last_clean":"2024-03-29T04:08:55.925452+0000","last_became_active":"2024-03-29T04:08:53.829295+0000","last_became_peered":"2024-03-29T04:08:53.829295+0000","last_unstale":"2024-03-29T04:08:55.925452+0000","last_undegraded":"2024-03-29T04:08:55.925452+0000","last_fullsized":"2024-03-29T04:08:55.925452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924536+0000","last_change":"2024-03-29T04:08:53.819700+0000","last_active":"2024-03-29T04:08:55.924536+0000","last_peered":"2024-03-29T04:08:55.924536+0000","last_clean":"2024-03-29T04:08:55.924536+0000","last_became_active":"2024-03-29T04:08:53.819174+0000","last_became_peered":"2024-03-29T04:08:53.819174+0000","last_unstale":"2024-03-29T04:08:55.924536+0000","last_undegraded":"2024-03-29T04:08:55.924536+0000","last_fullsized":"2024-03-29T04:08:55.924536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811825+0000","last_change":"2024-03-29T04:08:53.812928+0000","last_active":"2024-03-29T04:08:54.811825+0000","last_peered":"2024-03-29T04:08:54.811825+0000","last_clean":"2024-03-29T04:08:54.811825+0000","last_became_active":"2024-03-29T04:08:53.812536+0000","last_became_peered":"2024-03-29T04:08:53.812536+0000","last_unstale":"2024-03-29T04:08:54.811825+0000","last_undegraded":"2024-03-29T04:08:54.811825+0000","last_fullsized":"2024-03-29T04:08:54.811825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814066+0000","last_change":"2024-03-29T04:08:53.823468+0000","last_active":"2024-03-29T04:08:54.814066+0000","last_peered":"2024-03-29T04:08:54.814066+0000","last_clean":"2024-03-29T04:08:54.814066+0000","last_became_active":"2024-03-29T04:08:53.822972+0000","last_became_peered":"2024-03-29T04:08:53.822972+0000","last_unstale":"2024-03-29T04:08:54.814066+0000","last_undegraded":"2024-03-29T04:08:54.814066+0000","last_fullsized":"2024-03-29T04:08:54.814066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815690+0000","last_change":"2024-03-29T04:08:53.816411+0000","last_active":"2024-03-29T04:08:54.815690+0000","last_peered":"2024-03-29T04:08:54.815690+0000","last_clean":"2024-03-29T04:08:54.815690+0000","last_became_active":"2024-03-29T04:08:53.816126+0000","last_became_peered":"2024-03-29T04:08:53.816126+0000","last_unstale":"2024-03-29T04:08:54.815690+0000","last_undegraded":"2024-03-29T04:08:54.815690+0000","last_fullsized":"2024-03-29T04:08:54.815690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924748+0000","last_change":"2024-03-29T04:08:53.824360+0000","last_active":"2024-03-29T04:08:55.924748+0000","last_peered":"2024-03-29T04:08:55.924748+0000","last_clean":"2024-03-29T04:08:55.924748+0000","last_became_active":"2024-03-29T04:08:53.823207+0000","last_became_peered":"2024-03-29T04:08:53.823207+0000","last_unstale":"2024-03-29T04:08:55.924748+0000","last_undegraded":"2024-03-29T04:08:55.924748+0000","last_fullsized":"2024-03-29T04:08:55.924748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546389+0000","last_change":"2024-03-29T04:08:53.822584+0000","last_active":"2024-03-29T04:08:55.546389+0000","last_peered":"2024-03-29T04:08:55.546389+0000","last_clean":"2024-03-29T04:08:55.546389+0000","last_became_active":"2024-03-29T04:08:53.821230+0000","last_became_peered":"2024-03-29T04:08:53.821230+0000","last_unstale":"2024-03-29T04:08:55.546389+0000","last_undegraded":"2024-03-29T04:08:55.546389+0000","last_fullsized":"2024-03-29T04:08:55.546389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:16.297217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816186+0000","last_change":"2024-03-29T04:08:53.828332+0000","last_active":"2024-03-29T04:08:54.816186+0000","last_peered":"2024-03-29T04:08:54.816186+0000","last_clean":"2024-03-29T04:08:54.816186+0000","last_became_active":"2024-03-29T04:08:53.827954+0000","last_became_peered":"2024-03-29T04:08:53.827954+0000","last_unstale":"2024-03-29T04:08:54.816186+0000","last_undegraded":"2024-03-29T04:08:54.816186+0000","last_fullsized":"2024-03-29T04:08:54.816186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924504+0000","last_change":"2024-03-29T04:08:53.821853+0000","last_active":"2024-03-29T04:08:55.924504+0000","last_peered":"2024-03-29T04:08:55.924504+0000","last_clean":"2024-03-29T04:08:55.924504+0000","last_became_active":"2024-03-29T04:08:53.820469+0000","last_became_peered":"2024-03-29T04:08:53.820469+0000","last_unstale":"2024-03-29T04:08:55.924504+0000","last_undegraded":"2024-03-29T04:08:55.924504+0000","last_fullsized":"2024-03-29T04:08:55.924504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814489+0000","last_change":"2024-03-29T04:08:53.818641+0000","last_active":"2024-03-29T04:08:54.814489+0000","last_peered":"2024-03-29T04:08:54.814489+0000","last_clean":"2024-03-29T04:08:54.814489+0000","last_became_active":"2024-03-29T04:08:53.818082+0000","last_became_peered":"2024-03-29T04:08:53.818082+0000","last_unstale":"2024-03-29T04:08:54.814489+0000","last_undegraded":"2024-03-29T04:08:54.814489+0000","last_fullsized":"2024-03-29T04:08:54.814489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:56.736275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925283+0000","last_change":"2024-03-29T04:08:53.816408+0000","last_active":"2024-03-29T04:08:55.925283+0000","last_peered":"2024-03-29T04:08:55.925283+0000","last_clean":"2024-03-29T04:08:55.925283+0000","last_became_active":"2024-03-29T04:08:53.816127+0000","last_became_peered":"2024-03-29T04:08:53.816127+0000","last_unstale":"2024-03-29T04:08:55.925283+0000","last_undegraded":"2024-03-29T04:08:55.925283+0000","last_fullsized":"2024-03-29T04:08:55.925283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814214+0000","last_change":"2024-03-29T04:08:53.820680+0000","last_active":"2024-03-29T04:08:54.814214+0000","last_peered":"2024-03-29T04:08:54.814214+0000","last_clean":"2024-03-29T04:08:54.814214+0000","last_became_active":"2024-03-29T04:08:53.819878+0000","last_became_peered":"2024-03-29T04:08:53.819878+0000","last_unstale":"2024-03-29T04:08:54.814214+0000","last_undegraded":"2024-03-29T04:08:54.814214+0000","last_fullsized":"2024-03-29T04:08:54.814214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925741+0000","last_change":"2024-03-29T04:08:53.816044+0000","last_active":"2024-03-29T04:08:55.925741+0000","last_peered":"2024-03-29T04:08:55.925741+0000","last_clean":"2024-03-29T04:08:55.925741+0000","last_became_active":"2024-03-29T04:08:53.815744+0000","last_became_peered":"2024-03-29T04:08:53.815744+0000","last_unstale":"2024-03-29T04:08:55.925741+0000","last_undegraded":"2024-03-29T04:08:55.925741+0000","last_fullsized":"2024-03-29T04:08:55.925741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816521+0000","last_change":"2024-03-29T04:08:53.830594+0000","last_active":"2024-03-29T04:08:54.816521+0000","last_peered":"2024-03-29T04:08:54.816521+0000","last_clean":"2024-03-29T04:08:54.816521+0000","last_became_active":"2024-03-29T04:08:53.830266+0000","last_became_peered":"2024-03-29T04:08:53.830266+0000","last_unstale":"2024-03-29T04:08:54.816521+0000","last_undegraded":"2024-03-29T04:08:54.816521+0000","last_fullsized":"2024-03-29T04:08:54.816521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812076+0000","last_change":"2024-03-29T04:08:53.817458+0000","last_active":"2024-03-29T04:08:54.812076+0000","last_peered":"2024-03-29T04:08:54.812076+0000","last_clean":"2024-03-29T04:08:54.812076+0000","last_became_active":"2024-03-29T04:08:53.817179+0000","last_became_peered":"2024-03-29T04:08:53.817179+0000","last_unstale":"2024-03-29T04:08:54.812076+0000","last_undegraded":"2024-03-29T04:08:54.812076+0000","last_fullsized":"2024-03-29T04:08:54.812076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925638+0000","last_change":"2024-03-29T04:08:53.819794+0000","last_active":"2024-03-29T04:08:55.925638+0000","last_peered":"2024-03-29T04:08:55.925638+0000","last_clean":"2024-03-29T04:08:55.925638+0000","last_became_active":"2024-03-29T04:08:53.819541+0000","last_became_peered":"2024-03-29T04:08:53.819541+0000","last_unstale":"2024-03-29T04:08:55.925638+0000","last_undegraded":"2024-03-29T04:08:55.925638+0000","last_fullsized":"2024-03-29T04:08:55.925638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:34.960134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924635+0000","last_change":"2024-03-29T04:08:53.823205+0000","last_active":"2024-03-29T04:08:55.924635+0000","last_peered":"2024-03-29T04:08:55.924635+0000","last_clean":"2024-03-29T04:08:55.924635+0000","last_became_active":"2024-03-29T04:08:53.822900+0000","last_became_peered":"2024-03-29T04:08:53.822900+0000","last_unstale":"2024-03-29T04:08:55.924635+0000","last_undegraded":"2024-03-29T04:08:55.924635+0000","last_fullsized":"2024-03-29T04:08:55.924635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814624+0000","last_change":"2024-03-29T04:08:53.821709+0000","last_active":"2024-03-29T04:08:54.814624+0000","last_peered":"2024-03-29T04:08:54.814624+0000","last_clean":"2024-03-29T04:08:54.814624+0000","last_became_active":"2024-03-29T04:08:53.820828+0000","last_became_peered":"2024-03-29T04:08:53.820828+0000","last_unstale":"2024-03-29T04:08:54.814624+0000","last_undegraded":"2024-03-29T04:08:54.814624+0000","last_fullsized":"2024-03-29T04:08:54.814624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924175+0000","last_change":"2024-03-29T04:08:53.820303+0000","last_active":"2024-03-29T04:08:55.924175+0000","last_peered":"2024-03-29T04:08:55.924175+0000","last_clean":"2024-03-29T04:08:55.924175+0000","last_became_active":"2024-03-29T04:08:53.820024+0000","last_became_peered":"2024-03-29T04:08:53.820024+0000","last_unstale":"2024-03-29T04:08:55.924175+0000","last_undegraded":"2024-03-29T04:08:55.924175+0000","last_fullsized":"2024-03-29T04:08:55.924175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815813+0000","last_change":"2024-03-29T04:08:53.824803+0000","last_active":"2024-03-29T04:08:54.815813+0000","last_peered":"2024-03-29T04:08:54.815813+0000","last_clean":"2024-03-29T04:08:54.815813+0000","last_became_active":"2024-03-29T04:08:53.824454+0000","last_became_peered":"2024-03-29T04:08:53.824454+0000","last_unstale":"2024-03-29T04:08:54.815813+0000","last_undegraded":"2024-03-29T04:08:54.815813+0000","last_fullsized":"2024-03-29T04:08:54.815813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813758+0000","last_change":"2024-03-29T04:08:53.819297+0000","last_active":"2024-03-29T04:08:54.813758+0000","last_peered":"2024-03-29T04:08:54.813758+0000","last_clean":"2024-03-29T04:08:54.813758+0000","last_became_active":"2024-03-29T04:08:53.819016+0000","last_became_peered":"2024-03-29T04:08:53.819016+0000","last_unstale":"2024-03-29T04:08:54.813758+0000","last_undegraded":"2024-03-29T04:08:54.813758+0000","last_fullsized":"2024-03-29T04:08:54.813758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:03.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925546+0000","last_change":"2024-03-29T04:08:52.815351+0000","last_active":"2024-03-29T04:08:55.925546+0000","last_peered":"2024-03-29T04:08:55.925546+0000","last_clean":"2024-03-29T04:08:55.925546+0000","last_became_active":"2024-03-29T04:08:52.814942+0000","last_became_peered":"2024-03-29T04:08:52.814942+0000","last_unstale":"2024-03-29T04:08:55.925546+0000","last_undegraded":"2024-03-29T04:08:55.925546+0000","last_fullsized":"2024-03-29T04:08:55.925546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924762+0000","last_change":"2024-03-29T04:08:53.820823+0000","last_active":"2024-03-29T04:08:55.924762+0000","last_peered":"2024-03-29T04:08:55.924762+0000","last_clean":"2024-03-29T04:08:55.924762+0000","last_became_active":"2024-03-29T04:08:53.820429+0000","last_became_peered":"2024-03-29T04:08:53.820429+0000","last_unstale":"2024-03-29T04:08:55.924762+0000","last_undegraded":"2024-03-29T04:08:55.924762+0000","last_fullsized":"2024-03-29T04:08:55.924762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815991+0000","last_change":"2024-03-29T04:08:52.812694+0000","last_active":"2024-03-29T04:08:54.815991+0000","last_peered":"2024-03-29T04:08:54.815991+0000","last_clean":"2024-03-29T04:08:54.815991+0000","last_became_active":"2024-03-29T04:08:52.812429+0000","last_became_peered":"2024-03-29T04:08:52.812429+0000","last_unstale":"2024-03-29T04:08:54.815991+0000","last_undegraded":"2024-03-29T04:08:54.815991+0000","last_fullsized":"2024-03-29T04:08:54.815991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925275+0000","last_change":"2024-03-29T04:08:52.809190+0000","last_active":"2024-03-29T04:08:55.925275+0000","last_peered":"2024-03-29T04:08:55.925275+0000","last_clean":"2024-03-29T04:08:55.925275+0000","last_became_active":"2024-03-29T04:08:52.808923+0000","last_became_peered":"2024-03-29T04:08:52.808923+0000","last_unstale":"2024-03-29T04:08:55.925275+0000","last_undegraded":"2024-03-29T04:08:55.925275+0000","last_fullsized":"2024-03-29T04:08:55.925275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924506+0000","last_change":"2024-03-29T04:08:53.828865+0000","last_active":"2024-03-29T04:08:55.924506+0000","last_peered":"2024-03-29T04:08:55.924506+0000","last_clean":"2024-03-29T04:08:55.924506+0000","last_became_active":"2024-03-29T04:08:53.828601+0000","last_became_peered":"2024-03-29T04:08:53.828601+0000","last_unstale":"2024-03-29T04:08:55.924506+0000","last_undegraded":"2024-03-29T04:08:55.924506+0000","last_fullsized":"2024-03-29T04:08:55.924506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814227+0000","last_change":"2024-03-29T04:08:52.810336+0000","last_active":"2024-03-29T04:08:54.814227+0000","last_peered":"2024-03-29T04:08:54.814227+0000","last_clean":"2024-03-29T04:08:54.814227+0000","last_became_active":"2024-03-29T04:08:52.810021+0000","last_became_peered":"2024-03-29T04:08:52.810021+0000","last_unstale":"2024-03-29T04:08:54.814227+0000","last_undegraded":"2024-03-29T04:08:54.814227+0000","last_fullsized":"2024-03-29T04:08:54.814227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811960+0000","last_change":"2024-03-29T04:08:53.817702+0000","last_active":"2024-03-29T04:08:54.811960+0000","last_peered":"2024-03-29T04:08:54.811960+0000","last_clean":"2024-03-29T04:08:54.811960+0000","last_became_active":"2024-03-29T04:08:53.817117+0000","last_became_peered":"2024-03-29T04:08:53.817117+0000","last_unstale":"2024-03-29T04:08:54.811960+0000","last_undegraded":"2024-03-29T04:08:54.811960+0000","last_fullsized":"2024-03-29T04:08:54.811960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816011+0000","last_change":"2024-03-29T04:08:52.812898+0000","last_active":"2024-03-29T04:08:54.816011+0000","last_peered":"2024-03-29T04:08:54.816011+0000","last_clean":"2024-03-29T04:08:54.816011+0000","last_became_active":"2024-03-29T04:08:52.811143+0000","last_became_peered":"2024-03-29T04:08:52.811143+0000","last_unstale":"2024-03-29T04:08:54.816011+0000","last_undegraded":"2024-03-29T04:08:54.816011+0000","last_fullsized":"2024-03-29T04:08:54.816011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813759+0000","last_change":"2024-03-29T04:08:53.815139+0000","last_active":"2024-03-29T04:08:54.813759+0000","last_peered":"2024-03-29T04:08:54.813759+0000","last_clean":"2024-03-29T04:08:54.813759+0000","last_became_active":"2024-03-29T04:08:53.814836+0000","last_became_peered":"2024-03-29T04:08:53.814836+0000","last_unstale":"2024-03-29T04:08:54.813759+0000","last_undegraded":"2024-03-29T04:08:54.813759+0000","last_fullsized":"2024-03-29T04:08:54.813759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813822+0000","last_change":"2024-03-29T04:08:52.812383+0000","last_active":"2024-03-29T04:08:54.813822+0000","last_peered":"2024-03-29T04:08:54.813822+0000","last_clean":"2024-03-29T04:08:54.813822+0000","last_became_active":"2024-03-29T04:08:52.810343+0000","last_became_peered":"2024-03-29T04:08:52.810343+0000","last_unstale":"2024-03-29T04:08:54.813822+0000","last_undegraded":"2024-03-29T04:08:54.813822+0000","last_fullsized":"2024-03-29T04:08:54.813822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924573+0000","last_change":"2024-03-29T04:08:53.821898+0000","last_active":"2024-03-29T04:08:55.924573+0000","last_peered":"2024-03-29T04:08:55.924573+0000","last_clean":"2024-03-29T04:08:55.924573+0000","last_became_active":"2024-03-29T04:08:53.821126+0000","last_became_peered":"2024-03-29T04:08:53.821126+0000","last_unstale":"2024-03-29T04:08:55.924573+0000","last_undegraded":"2024-03-29T04:08:55.924573+0000","last_fullsized":"2024-03-29T04:08:55.924573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812079+0000","last_change":"2024-03-29T04:08:52.810835+0000","last_active":"2024-03-29T04:08:54.812079+0000","last_peered":"2024-03-29T04:08:54.812079+0000","last_clean":"2024-03-29T04:08:54.812079+0000","last_became_active":"2024-03-29T04:08:52.810585+0000","last_became_peered":"2024-03-29T04:08:52.810585+0000","last_unstale":"2024-03-29T04:08:54.812079+0000","last_undegraded":"2024-03-29T04:08:54.812079+0000","last_fullsized":"2024-03-29T04:08:54.812079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814427+0000","last_change":"2024-03-29T04:08:53.817718+0000","last_active":"2024-03-29T04:08:54.814427+0000","last_peered":"2024-03-29T04:08:54.814427+0000","last_clean":"2024-03-29T04:08:54.814427+0000","last_became_active":"2024-03-29T04:08:53.817419+0000","last_became_peered":"2024-03-29T04:08:53.817419+0000","last_unstale":"2024-03-29T04:08:54.814427+0000","last_undegraded":"2024-03-29T04:08:54.814427+0000","last_fullsized":"2024-03-29T04:08:54.814427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:40.718195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816294+0000","last_change":"2024-03-29T04:08:52.809428+0000","last_active":"2024-03-29T04:08:54.816294+0000","last_peered":"2024-03-29T04:08:54.816294+0000","last_clean":"2024-03-29T04:08:54.816294+0000","last_became_active":"2024-03-29T04:08:52.809124+0000","last_became_peered":"2024-03-29T04:08:52.809124+0000","last_unstale":"2024-03-29T04:08:54.816294+0000","last_undegraded":"2024-03-29T04:08:54.816294+0000","last_fullsized":"2024-03-29T04:08:54.816294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546068+0000","last_change":"2024-03-29T04:08:53.823456+0000","last_active":"2024-03-29T04:08:55.546068+0000","last_peered":"2024-03-29T04:08:55.546068+0000","last_clean":"2024-03-29T04:08:55.546068+0000","last_became_active":"2024-03-29T04:08:53.823196+0000","last_became_peered":"2024-03-29T04:08:53.823196+0000","last_unstale":"2024-03-29T04:08:55.546068+0000","last_undegraded":"2024-03-29T04:08:55.546068+0000","last_fullsized":"2024-03-29T04:08:55.546068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924849+0000","last_change":"2024-03-29T04:08:52.804947+0000","last_active":"2024-03-29T04:08:55.924849+0000","last_peered":"2024-03-29T04:08:55.924849+0000","last_clean":"2024-03-29T04:08:55.924849+0000","last_became_active":"2024-03-29T04:08:52.804675+0000","last_became_peered":"2024-03-29T04:08:52.804675+0000","last_unstale":"2024-03-29T04:08:55.924849+0000","last_undegraded":"2024-03-29T04:08:55.924849+0000","last_fullsized":"2024-03-29T04:08:55.924849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:05.363292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924675+0000","last_change":"2024-03-29T04:08:53.815297+0000","last_active":"2024-03-29T04:08:55.924675+0000","last_peered":"2024-03-29T04:08:55.924675+0000","last_clean":"2024-03-29T04:08:55.924675+0000","last_became_active":"2024-03-29T04:08:53.815003+0000","last_became_peered":"2024-03-29T04:08:53.815003+0000","last_unstale":"2024-03-29T04:08:55.924675+0000","last_undegraded":"2024-03-29T04:08:55.924675+0000","last_fullsized":"2024-03-29T04:08:55.924675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.423583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814596+0000","last_change":"2024-03-29T04:08:52.809942+0000","last_active":"2024-03-29T04:08:54.814596+0000","last_peered":"2024-03-29T04:08:54.814596+0000","last_clean":"2024-03-29T04:08:54.814596+0000","last_became_active":"2024-03-29T04:08:52.809671+0000","last_became_peered":"2024-03-29T04:08:52.809671+0000","last_unstale":"2024-03-29T04:08:54.814596+0000","last_undegraded":"2024-03-29T04:08:54.814596+0000","last_fullsized":"2024-03-29T04:08:54.814596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546150+0000","last_change":"2024-03-29T04:08:53.818585+0000","last_active":"2024-03-29T04:08:55.546150+0000","last_peered":"2024-03-29T04:08:55.546150+0000","last_clean":"2024-03-29T04:08:55.546150+0000","last_became_active":"2024-03-29T04:08:53.818252+0000","last_became_peered":"2024-03-29T04:08:53.818252+0000","last_unstale":"2024-03-29T04:08:55.546150+0000","last_undegraded":"2024-03-29T04:08:55.546150+0000","last_fullsized":"2024-03-29T04:08:55.546150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:50.957059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815950+0000","last_change":"2024-03-29T04:08:52.810780+0000","last_active":"2024-03-29T04:08:54.815950+0000","last_peered":"2024-03-29T04:08:54.815950+0000","last_clean":"2024-03-29T04:08:54.815950+0000","last_became_active":"2024-03-29T04:08:52.810487+0000","last_became_peered":"2024-03-29T04:08:52.810487+0000","last_unstale":"2024-03-29T04:08:54.815950+0000","last_undegraded":"2024-03-29T04:08:54.815950+0000","last_fullsized":"2024-03-29T04:08:54.815950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816134+0000","last_change":"2024-03-29T04:08:53.825665+0000","last_active":"2024-03-29T04:08:54.816134+0000","last_peered":"2024-03-29T04:08:54.816134+0000","last_clean":"2024-03-29T04:08:54.816134+0000","last_became_active":"2024-03-29T04:08:53.825229+0000","last_became_peered":"2024-03-29T04:08:53.825229+0000","last_unstale":"2024-03-29T04:08:54.816134+0000","last_undegraded":"2024-03-29T04:08:54.816134+0000","last_fullsized":"2024-03-29T04:08:54.816134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:28.277522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924756+0000","last_change":"2024-03-29T04:08:52.807491+0000","last_active":"2024-03-29T04:08:55.924756+0000","last_peered":"2024-03-29T04:08:55.924756+0000","last_clean":"2024-03-29T04:08:55.924756+0000","last_became_active":"2024-03-29T04:08:52.807236+0000","last_became_peered":"2024-03-29T04:08:52.807236+0000","last_unstale":"2024-03-29T04:08:55.924756+0000","last_undegraded":"2024-03-29T04:08:55.924756+0000","last_fullsized":"2024-03-29T04:08:55.924756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545849+0000","last_change":"2024-03-29T04:08:53.819664+0000","last_active":"2024-03-29T04:08:55.545849+0000","last_peered":"2024-03-29T04:08:55.545849+0000","last_clean":"2024-03-29T04:08:55.545849+0000","last_became_active":"2024-03-29T04:08:53.819414+0000","last_became_peered":"2024-03-29T04:08:53.819414+0000","last_unstale":"2024-03-29T04:08:55.545849+0000","last_undegraded":"2024-03-29T04:08:55.545849+0000","last_fullsized":"2024-03-29T04:08:55.545849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:21.843106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815063+0000","last_change":"2024-03-29T04:08:52.806445+0000","last_active":"2024-03-29T04:08:54.815063+0000","last_peered":"2024-03-29T04:08:54.815063+0000","last_clean":"2024-03-29T04:08:54.815063+0000","last_became_active":"2024-03-29T04:08:52.806166+0000","last_became_peered":"2024-03-29T04:08:52.806166+0000","last_unstale":"2024-03-29T04:08:54.815063+0000","last_undegraded":"2024-03-29T04:08:54.815063+0000","last_fullsized":"2024-03-29T04:08:54.815063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:03.546395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546168+0000","last_change":"2024-03-29T04:08:53.819783+0000","last_active":"2024-03-29T04:08:55.546168+0000","last_peered":"2024-03-29T04:08:55.546168+0000","last_clean":"2024-03-29T04:08:55.546168+0000","last_became_active":"2024-03-29T04:08:53.819507+0000","last_became_peered":"2024-03-29T04:08:53.819507+0000","last_unstale":"2024-03-29T04:08:55.546168+0000","last_undegraded":"2024-03-29T04:08:55.546168+0000","last_fullsized":"2024-03-29T04:08:55.546168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:32:01.627691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924833+0000","last_change":"2024-03-29T04:08:52.812349+0000","last_active":"2024-03-29T04:08:55.924833+0000","last_peered":"2024-03-29T04:08:55.924833+0000","last_clean":"2024-03-29T04:08:55.924833+0000","last_became_active":"2024-03-29T04:08:52.812081+0000","last_became_peered":"2024-03-29T04:08:52.812081+0000","last_unstale":"2024-03-29T04:08:55.924833+0000","last_undegraded":"2024-03-29T04:08:55.924833+0000","last_fullsized":"2024-03-29T04:08:55.924833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546387+0000","last_change":"2024-03-29T04:08:53.827351+0000","last_active":"2024-03-29T04:08:55.546387+0000","last_peered":"2024-03-29T04:08:55.546387+0000","last_clean":"2024-03-29T04:08:55.546387+0000","last_became_active":"2024-03-29T04:08:53.826255+0000","last_became_peered":"2024-03-29T04:08:53.826255+0000","last_unstale":"2024-03-29T04:08:55.546387+0000","last_undegraded":"2024-03-29T04:08:55.546387+0000","last_fullsized":"2024-03-29T04:08:55.546387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812164+0000","last_change":"2024-03-29T04:08:52.805752+0000","last_active":"2024-03-29T04:08:54.812164+0000","last_peered":"2024-03-29T04:08:54.812164+0000","last_clean":"2024-03-29T04:08:54.812164+0000","last_became_active":"2024-03-29T04:08:52.804921+0000","last_became_peered":"2024-03-29T04:08:52.804921+0000","last_unstale":"2024-03-29T04:08:54.812164+0000","last_undegraded":"2024-03-29T04:08:54.812164+0000","last_fullsized":"2024-03-29T04:08:54.812164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815243+0000","last_change":"2024-03-29T04:08:53.821399+0000","last_active":"2024-03-29T04:08:54.815243+0000","last_peered":"2024-03-29T04:08:54.815243+0000","last_clean":"2024-03-29T04:08:54.815243+0000","last_became_active":"2024-03-29T04:08:53.821026+0000","last_became_peered":"2024-03-29T04:08:53.821026+0000","last_unstale":"2024-03-29T04:08:54.815243+0000","last_undegraded":"2024-03-29T04:08:54.815243+0000","last_fullsized":"2024-03-29T04:08:54.815243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:02.718680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924593+0000","last_change":"2024-03-29T04:08:52.811397+0000","last_active":"2024-03-29T04:08:55.924593+0000","last_peered":"2024-03-29T04:08:55.924593+0000","last_clean":"2024-03-29T04:08:55.924593+0000","last_became_active":"2024-03-29T04:08:52.810968+0000","last_became_peered":"2024-03-29T04:08:52.810968+0000","last_unstale":"2024-03-29T04:08:55.924593+0000","last_undegraded":"2024-03-29T04:08:55.924593+0000","last_fullsized":"2024-03-29T04:08:55.924593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.812910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546224+0000","last_change":"2024-03-29T04:08:53.824444+0000","last_active":"2024-03-29T04:08:55.546224+0000","last_peered":"2024-03-29T04:08:55.546224+0000","last_clean":"2024-03-29T04:08:55.546224+0000","last_became_active":"2024-03-29T04:08:53.821578+0000","last_became_peered":"2024-03-29T04:08:53.821578+0000","last_unstale":"2024-03-29T04:08:55.546224+0000","last_undegraded":"2024-03-29T04:08:55.546224+0000","last_fullsized":"2024-03-29T04:08:55.546224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545812+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:08:55.545812+0000","last_peered":"2024-03-29T04:08:55.545812+0000","last_clean":"2024-03-29T04:08:55.545812+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:08:55.545812+0000","last_undegraded":"2024-03-29T04:08:55.545812+0000","last_fullsized":"2024-03-29T04:08:55.545812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.813649+0000","last_change":"2024-03-29T04:08:52.807738+0000","last_active":"2024-03-29T04:08:54.813649+0000","last_peered":"2024-03-29T04:08:54.813649+0000","last_clean":"2024-03-29T04:08:54.813649+0000","last_became_active":"2024-03-29T04:08:52.807442+0000","last_became_peered":"2024-03-29T04:08:52.807442+0000","last_unstale":"2024-03-29T04:08:54.813649+0000","last_undegraded":"2024-03-29T04:08:54.813649+0000","last_fullsized":"2024-03-29T04:08:54.813649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925459+0000","last_change":"2024-03-29T04:08:53.825630+0000","last_active":"2024-03-29T04:08:55.925459+0000","last_peered":"2024-03-29T04:08:55.925459+0000","last_clean":"2024-03-29T04:08:55.925459+0000","last_became_active":"2024-03-29T04:08:53.825386+0000","last_became_peered":"2024-03-29T04:08:53.825386+0000","last_unstale":"2024-03-29T04:08:55.925459+0000","last_undegraded":"2024-03-29T04:08:55.925459+0000","last_fullsized":"2024-03-29T04:08:55.925459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545708+0000","last_change":"2024-03-29T04:08:52.820187+0000","last_active":"2024-03-29T04:08:55.545708+0000","last_peered":"2024-03-29T04:08:55.545708+0000","last_clean":"2024-03-29T04:08:55.545708+0000","last_became_active":"2024-03-29T04:08:52.819912+0000","last_became_peered":"2024-03-29T04:08:52.819912+0000","last_unstale":"2024-03-29T04:08:55.545708+0000","last_undegraded":"2024-03-29T04:08:55.545708+0000","last_fullsized":"2024-03-29T04:08:55.545708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924298+0000","last_change":"2024-03-29T04:08:53.815607+0000","last_active":"2024-03-29T04:08:55.924298+0000","last_peered":"2024-03-29T04:08:55.924298+0000","last_clean":"2024-03-29T04:08:55.924298+0000","last_became_active":"2024-03-29T04:08:53.814274+0000","last_became_peered":"2024-03-29T04:08:53.814274+0000","last_unstale":"2024-03-29T04:08:55.924298+0000","last_undegraded":"2024-03-29T04:08:55.924298+0000","last_fullsized":"2024-03-29T04:08:55.924298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:39.590764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924274+0000","last_change":"2024-03-29T04:08:52.813143+0000","last_active":"2024-03-29T04:08:55.924274+0000","last_peered":"2024-03-29T04:08:55.924274+0000","last_clean":"2024-03-29T04:08:55.924274+0000","last_became_active":"2024-03-29T04:08:52.812866+0000","last_became_peered":"2024-03-29T04:08:52.812866+0000","last_unstale":"2024-03-29T04:08:55.924274+0000","last_undegraded":"2024-03-29T04:08:55.924274+0000","last_fullsized":"2024-03-29T04:08:55.924274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.897788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811826+0000","last_change":"2024-03-29T04:08:53.818488+0000","last_active":"2024-03-29T04:08:54.811826+0000","last_peered":"2024-03-29T04:08:54.811826+0000","last_clean":"2024-03-29T04:08:54.811826+0000","last_became_active":"2024-03-29T04:08:53.818222+0000","last_became_peered":"2024-03-29T04:08:53.818222+0000","last_unstale":"2024-03-29T04:08:54.811826+0000","last_undegraded":"2024-03-29T04:08:54.811826+0000","last_fullsized":"2024-03-29T04:08:54.811826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:38.595764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814714+0000","last_change":"2024-03-29T04:08:52.808830+0000","last_active":"2024-03-29T04:08:54.814714+0000","last_peered":"2024-03-29T04:08:54.814714+0000","last_clean":"2024-03-29T04:08:54.814714+0000","last_became_active":"2024-03-29T04:08:52.808524+0000","last_became_peered":"2024-03-29T04:08:52.808524+0000","last_unstale":"2024-03-29T04:08:54.814714+0000","last_undegraded":"2024-03-29T04:08:54.814714+0000","last_fullsized":"2024-03-29T04:08:54.814714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815259+0000","last_change":"2024-03-29T04:08:53.823062+0000","last_active":"2024-03-29T04:08:54.815259+0000","last_peered":"2024-03-29T04:08:54.815259+0000","last_clean":"2024-03-29T04:08:54.815259+0000","last_became_active":"2024-03-29T04:08:53.822666+0000","last_became_peered":"2024-03-29T04:08:53.822666+0000","last_unstale":"2024-03-29T04:08:54.815259+0000","last_undegraded":"2024-03-29T04:08:54.815259+0000","last_fullsized":"2024-03-29T04:08:54.815259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:37.672885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812019+0000","last_change":"2024-03-29T04:08:52.799226+0000","last_active":"2024-03-29T04:08:54.812019+0000","last_peered":"2024-03-29T04:08:54.812019+0000","last_clean":"2024-03-29T04:08:54.812019+0000","last_became_active":"2024-03-29T04:08:52.798964+0000","last_became_peered":"2024-03-29T04:08:52.798964+0000","last_unstale":"2024-03-29T04:08:54.812019+0000","last_undegraded":"2024-03-29T04:08:54.812019+0000","last_fullsized":"2024-03-29T04:08:54.812019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815451+0000","last_change":"2024-03-29T04:08:53.821737+0000","last_active":"2024-03-29T04:08:54.815451+0000","last_peered":"2024-03-29T04:08:54.815451+0000","last_clean":"2024-03-29T04:08:54.815451+0000","last_became_active":"2024-03-29T04:08:53.820680+0000","last_became_peered":"2024-03-29T04:08:53.820680+0000","last_unstale":"2024-03-29T04:08:54.815451+0000","last_undegraded":"2024-03-29T04:08:54.815451+0000","last_fullsized":"2024-03-29T04:08:54.815451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:49.498846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546333+0000","last_change":"2024-03-29T04:08:52.821118+0000","last_active":"2024-03-29T04:08:55.546333+0000","last_peered":"2024-03-29T04:08:55.546333+0000","last_clean":"2024-03-29T04:08:55.546333+0000","last_became_active":"2024-03-29T04:08:52.820549+0000","last_became_peered":"2024-03-29T04:08:52.820549+0000","last_unstale":"2024-03-29T04:08:55.546333+0000","last_undegraded":"2024-03-29T04:08:55.546333+0000","last_fullsized":"2024-03-29T04:08:55.546333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811978+0000","last_change":"2024-03-29T04:08:53.810781+0000","last_active":"2024-03-29T04:08:54.811978+0000","last_peered":"2024-03-29T04:08:54.811978+0000","last_clean":"2024-03-29T04:08:54.811978+0000","last_became_active":"2024-03-29T04:08:53.810518+0000","last_became_peered":"2024-03-29T04:08:53.810518+0000","last_unstale":"2024-03-29T04:08:54.811978+0000","last_undegraded":"2024-03-29T04:08:54.811978+0000","last_fullsized":"2024-03-29T04:08:54.811978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:27.701669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816447+0000","last_change":"2024-03-29T04:08:52.809876+0000","last_active":"2024-03-29T04:08:54.816447+0000","last_peered":"2024-03-29T04:08:54.816447+0000","last_clean":"2024-03-29T04:08:54.816447+0000","last_became_active":"2024-03-29T04:08:52.809575+0000","last_became_peered":"2024-03-29T04:08:52.809575+0000","last_unstale":"2024-03-29T04:08:54.816447+0000","last_undegraded":"2024-03-29T04:08:54.816447+0000","last_fullsized":"2024-03-29T04:08:54.816447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925771+0000","last_change":"2024-03-29T04:08:53.831225+0000","last_active":"2024-03-29T04:08:55.925771+0000","last_peered":"2024-03-29T04:08:55.925771+0000","last_clean":"2024-03-29T04:08:55.925771+0000","last_became_active":"2024-03-29T04:08:53.830819+0000","last_became_peered":"2024-03-29T04:08:53.830819+0000","last_unstale":"2024-03-29T04:08:55.925771+0000","last_undegraded":"2024-03-29T04:08:55.925771+0000","last_fullsized":"2024-03-29T04:08:55.925771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:10.297183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.816132+0000","last_change":"2024-03-29T04:08:52.812725+0000","last_active":"2024-03-29T04:08:54.816132+0000","last_peered":"2024-03-29T04:08:54.816132+0000","last_clean":"2024-03-29T04:08:54.816132+0000","last_became_active":"2024-03-29T04:08:52.812020+0000","last_became_peered":"2024-03-29T04:08:52.812020+0000","last_unstale":"2024-03-29T04:08:54.816132+0000","last_undegraded":"2024-03-29T04:08:54.816132+0000","last_fullsized":"2024-03-29T04:08:54.816132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:19:17.992006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925009+0000","last_change":"2024-03-29T04:08:53.811809+0000","last_active":"2024-03-29T04:08:55.925009+0000","last_peered":"2024-03-29T04:08:55.925009+0000","last_clean":"2024-03-29T04:08:55.925009+0000","last_became_active":"2024-03-29T04:08:53.811513+0000","last_became_peered":"2024-03-29T04:08:53.811513+0000","last_unstale":"2024-03-29T04:08:55.925009+0000","last_undegraded":"2024-03-29T04:08:55.925009+0000","last_fullsized":"2024-03-29T04:08:55.925009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545699+0000","last_change":"2024-03-29T04:08:52.814972+0000","last_active":"2024-03-29T04:08:55.545699+0000","last_peered":"2024-03-29T04:08:55.545699+0000","last_clean":"2024-03-29T04:08:55.545699+0000","last_became_active":"2024-03-29T04:08:52.814714+0000","last_became_peered":"2024-03-29T04:08:52.814714+0000","last_unstale":"2024-03-29T04:08:55.545699+0000","last_undegraded":"2024-03-29T04:08:55.545699+0000","last_fullsized":"2024-03-29T04:08:55.545699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:02.617715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.814365+0000","last_change":"2024-03-29T04:08:53.817677+0000","last_active":"2024-03-29T04:08:54.814365+0000","last_peered":"2024-03-29T04:08:54.814365+0000","last_clean":"2024-03-29T04:08:54.814365+0000","last_became_active":"2024-03-29T04:08:53.815481+0000","last_became_peered":"2024-03-29T04:08:53.815481+0000","last_unstale":"2024-03-29T04:08:54.814365+0000","last_undegraded":"2024-03-29T04:08:54.814365+0000","last_fullsized":"2024-03-29T04:08:54.814365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:02.322751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.811927+0000","last_change":"2024-03-29T04:08:52.814031+0000","last_active":"2024-03-29T04:08:54.811927+0000","last_peered":"2024-03-29T04:08:54.811927+0000","last_clean":"2024-03-29T04:08:54.811927+0000","last_became_active":"2024-03-29T04:08:52.813714+0000","last_became_peered":"2024-03-29T04:08:52.813714+0000","last_unstale":"2024-03-29T04:08:54.811927+0000","last_undegraded":"2024-03-29T04:08:54.811927+0000","last_fullsized":"2024-03-29T04:08:54.811927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:36.668906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924641+0000","last_change":"2024-03-29T04:08:53.818653+0000","last_active":"2024-03-29T04:08:55.924641+0000","last_peered":"2024-03-29T04:08:55.924641+0000","last_clean":"2024-03-29T04:08:55.924641+0000","last_became_active":"2024-03-29T04:08:53.818387+0000","last_became_peered":"2024-03-29T04:08:53.818387+0000","last_unstale":"2024-03-29T04:08:55.924641+0000","last_undegraded":"2024-03-29T04:08:55.924641+0000","last_fullsized":"2024-03-29T04:08:55.924641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:06.088474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.812062+0000","last_change":"2024-03-29T04:08:52.806456+0000","last_active":"2024-03-29T04:08:54.812062+0000","last_peered":"2024-03-29T04:08:54.812062+0000","last_clean":"2024-03-29T04:08:54.812062+0000","last_became_active":"2024-03-29T04:08:52.806184+0000","last_became_peered":"2024-03-29T04:08:52.806184+0000","last_unstale":"2024-03-29T04:08:54.812062+0000","last_undegraded":"2024-03-29T04:08:54.812062+0000","last_fullsized":"2024-03-29T04:08:54.812062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925468+0000","last_change":"2024-03-29T04:08:53.828867+0000","last_active":"2024-03-29T04:08:55.925468+0000","last_peered":"2024-03-29T04:08:55.925468+0000","last_clean":"2024-03-29T04:08:55.925468+0000","last_became_active":"2024-03-29T04:08:53.828610+0000","last_became_peered":"2024-03-29T04:08:53.828610+0000","last_unstale":"2024-03-29T04:08:55.925468+0000","last_undegraded":"2024-03-29T04:08:55.925468+0000","last_fullsized":"2024-03-29T04:08:55.925468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:23.076492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546226+0000","last_change":"2024-03-29T04:08:52.821021+0000","last_active":"2024-03-29T04:08:55.546226+0000","last_peered":"2024-03-29T04:08:55.546226+0000","last_clean":"2024-03-29T04:08:55.546226+0000","last_became_active":"2024-03-29T04:08:52.820478+0000","last_became_peered":"2024-03-29T04:08:52.820478+0000","last_unstale":"2024-03-29T04:08:55.546226+0000","last_undegraded":"2024-03-29T04:08:55.546226+0000","last_fullsized":"2024-03-29T04:08:55.546226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:47:21.517361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815868+0000","last_change":"2024-03-29T04:08:53.825032+0000","last_active":"2024-03-29T04:08:54.815868+0000","last_peered":"2024-03-29T04:08:54.815868+0000","last_clean":"2024-03-29T04:08:54.815868+0000","last_became_active":"2024-03-29T04:08:53.824740+0000","last_became_peered":"2024-03-29T04:08:53.824740+0000","last_unstale":"2024-03-29T04:08:54.815868+0000","last_undegraded":"2024-03-29T04:08:54.815868+0000","last_fullsized":"2024-03-29T04:08:54.815868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:51.757357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815425+0000","last_change":"2024-03-29T04:08:53.816669+0000","last_active":"2024-03-29T04:08:54.815425+0000","last_peered":"2024-03-29T04:08:54.815425+0000","last_clean":"2024-03-29T04:08:54.815425+0000","last_became_active":"2024-03-29T04:08:53.816348+0000","last_became_peered":"2024-03-29T04:08:53.816348+0000","last_unstale":"2024-03-29T04:08:54.815425+0000","last_undegraded":"2024-03-29T04:08:54.815425+0000","last_fullsized":"2024-03-29T04:08:54.815425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:00.401420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.545977+0000","last_change":"2024-03-29T04:08:52.820909+0000","last_active":"2024-03-29T04:08:55.545977+0000","last_peered":"2024-03-29T04:08:55.545977+0000","last_clean":"2024-03-29T04:08:55.545977+0000","last_became_active":"2024-03-29T04:08:52.820523+0000","last_became_peered":"2024-03-29T04:08:52.820523+0000","last_unstale":"2024-03-29T04:08:55.545977+0000","last_undegraded":"2024-03-29T04:08:55.545977+0000","last_fullsized":"2024-03-29T04:08:55.545977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:01.318772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:54.815127+0000","last_change":"2024-03-29T04:08:53.817769+0000","last_active":"2024-03-29T04:08:54.815127+0000","last_peered":"2024-03-29T04:08:54.815127+0000","last_clean":"2024-03-29T04:08:54.815127+0000","last_became_active":"2024-03-29T04:08:53.816753+0000","last_became_peered":"2024-03-29T04:08:53.816753+0000","last_unstale":"2024-03-29T04:08:54.815127+0000","last_undegraded":"2024-03-29T04:08:54.815127+0000","last_fullsized":"2024-03-29T04:08:54.815127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:44.086463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.546206+0000","last_change":"2024-03-29T04:08:52.820167+0000","last_active":"2024-03-29T04:08:55.546206+0000","last_peered":"2024-03-29T04:08:55.546206+0000","last_clean":"2024-03-29T04:08:55.546206+0000","last_became_active":"2024-03-29T04:08:52.819858+0000","last_became_peered":"2024-03-29T04:08:52.819858+0000","last_unstale":"2024-03-29T04:08:55.546206+0000","last_undegraded":"2024-03-29T04:08:55.546206+0000","last_fullsized":"2024-03-29T04:08:55.546206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:33:39.714215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924413+0000","last_change":"2024-03-29T04:08:53.822001+0000","last_active":"2024-03-29T04:08:55.924413+0000","last_peered":"2024-03-29T04:08:55.924413+0000","last_clean":"2024-03-29T04:08:55.924413+0000","last_became_active":"2024-03-29T04:08:53.821669+0000","last_became_peered":"2024-03-29T04:08:53.821669+0000","last_unstale":"2024-03-29T04:08:55.924413+0000","last_undegraded":"2024-03-29T04:08:55.924413+0000","last_fullsized":"2024-03-29T04:08:55.924413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:19.825279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925665+0000","last_change":"2024-03-29T04:08:52.807347+0000","last_active":"2024-03-29T04:08:55.925665+0000","last_peered":"2024-03-29T04:08:55.925665+0000","last_clean":"2024-03-29T04:08:55.925665+0000","last_became_active":"2024-03-29T04:08:52.807042+0000","last_became_peered":"2024-03-29T04:08:52.807042+0000","last_unstale":"2024-03-29T04:08:55.925665+0000","last_undegraded":"2024-03-29T04:08:55.925665+0000","last_fullsized":"2024-03-29T04:08:55.925665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.925593+0000","last_change":"2024-03-29T04:08:53.829560+0000","last_active":"2024-03-29T04:08:55.925593+0000","last_peered":"2024-03-29T04:08:55.925593+0000","last_clean":"2024-03-29T04:08:55.925593+0000","last_became_active":"2024-03-29T04:08:53.829312+0000","last_became_peered":"2024-03-29T04:08:53.829312+0000","last_unstale":"2024-03-29T04:08:55.925593+0000","last_undegraded":"2024-03-29T04:08:55.925593+0000","last_fullsized":"2024-03-29T04:08:55.925593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:52.788734+0000","last_clean_scrub_stamp":"2024-03-29T04:08:52.788734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:30.140409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-29T04:08:55.924658+0000","last_change":"2024-03-29T04:08:52.813322+0000","last_active":"2024-03-29T04:08:55.924658+0000","last_peered":"2024-03-29T04:08:55.924658+0000","last_clean":"2024-03-29T04:08:55.924658+0000","last_became_active":"2024-03-29T04:08:52.813020+0000","last_became_peered":"2024-03-29T04:08:52.813020+0000","last_unstale":"2024-03-29T04:08:55.924658+0000","last_undegraded":"2024-03-29T04:08:55.924658+0000","last_fullsized":"2024-03-29T04:08:55.924658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:51.784754+0000","last_clean_scrub_stamp":"2024-03-29T04:08:51.784754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:57.197158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":665435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":225280,"data_stored":76097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":74485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":74645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":819200,"data_stored":665491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:09:03.817 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-29T04:09:03.817 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-29T04:09:03.817 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-29T04:09:03.817 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-29T04:09:03.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.972+0000 7f3387e41640 1 Processor -- start 2024-03-29T04:09:03.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.972+0000 7f3387e41640 1 -- start start 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f3387e41640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f33800cec00 0x7f33801609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f3387e41640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33800ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f3387e41640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33800cd1b0 con 0x7f33801568f0 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f3387e41640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f33800cf8b0 con 0x7f33800cec00 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f3387e41640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f33800cf0b0 con 0x7f33800cc870 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f33853dc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f33800cec00 0x7f33801609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.973+0000 7f3385bdd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f33801568f0 0x7f338015d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47542/0 (socket says 172.21.15.156:47542) 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f3385bdd640 1 -- 172.21.15.156:0/3963170999 learned_addr learned my addr 172.21.15.156:0/3963170999 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33853dc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f33800cec00 0x7f33801609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48536/0 (socket says 172.21.15.156:48536) 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33863de640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33800ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33853dc640 1 -- 172.21.15.156:0/3963170999 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 msgr2=0x7f33800ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:03.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33853dc640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33800ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33853dc640 1 -- 172.21.15.156:0/3963170999 >> v1:172.21.15.156:6789/0 conn(0x7f33801568f0 legacy=0x7f338015d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f3384bdb640 1 -- 172.21.15.156:0/3963170999 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2117099461 0 0) 0x7f33800cd1b0 con 0x7f33801568f0 2024-03-29T04:09:03.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33853dc640 1 -- 172.21.15.156:0/3963170999 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338015cb50 con 0x7f33800cec00 2024-03-29T04:09:03.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.974+0000 7f33863de640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33800ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:03.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.975+0000 7f33853dc640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f33801609b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f337c0097a0 tx=0x7f337c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f2a626f72c5344 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.975+0000 7f3384bdb640 1 -- 172.21.15.156:0/3963170999 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c0315a0 con 0x7f33800cec00 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.975+0000 7f3384bdb640 1 -- 172.21.15.156:0/3963170999 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f337c037070 con 0x7f33800cec00 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.975+0000 7f3384bdb640 1 -- 172.21.15.156:0/3963170999 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c031960 con 0x7f33800cec00 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 -- 172.21.15.156:0/3963170999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 msgr2=0x7f33801609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f33801609b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f337c0097a0 tx=0x7f337c02b840 comp rx=0 tx=0).stop 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 -- 172.21.15.156:0/3963170999 shutdown_connections 2024-03-29T04:09:03.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33800ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 --2- 172.21.15.156:0/3963170999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f33801609b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 -- 172.21.15.156:0/3963170999 >> 172.21.15.156:0/3963170999 conn(0x7f33800b3eb0 msgr2=0x7f33800b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:03.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 -- 172.21.15.156:0/3963170999 shutdown_connections 2024-03-29T04:09:03.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.976+0000 7f3387e41640 1 -- 172.21.15.156:0/3963170999 wait complete. 2024-03-29T04:09:03.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.977+0000 7f3387e41640 1 Processor -- start 2024-03-29T04:09:03.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.977+0000 7f3387e41640 1 -- start start 2024-03-29T04:09:03.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3387e41640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33801383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3387e41640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f338014a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3387e41640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33801568f0 0x7f338014aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3387e41640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f338015cdf0 con 0x7f33800cec00 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3387e41640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f33800cf5d0 con 0x7f33801568f0 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3387e41640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3380156cd0 con 0x7f33800cc870 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f338014a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f338014a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48550/0 (socket says 172.21.15.156:48550) 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 -- 172.21.15.156:0/647999257 learned_addr learned my addr 172.21.15.156:0/647999257 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f3385bdd640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33801383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33863de640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33801568f0 0x7f338014aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 -- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 msgr2=0x7f33801383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33801383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 -- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33801568f0 msgr2=0x7f338014aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.978+0000 7f33853dc640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33801568f0 0x7f338014aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f33853dc640 1 -- 172.21.15.156:0/647999257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3380139340 con 0x7f33800cec00 2024-03-29T04:09:03.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f33853dc640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f338014a500 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f337c0097a0 tx=0x7f337c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c004230 con 0x7f33800cec00 2024-03-29T04:09:03.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f337c0043f0 con 0x7f33800cec00 2024-03-29T04:09:03.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3380138d80 con 0x7f33800cec00 2024-03-29T04:09:03.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c032880 con 0x7f33800cec00 2024-03-29T04:09:03.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.979+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33800d1110 con 0x7f33800cec00 2024-03-29T04:09:03.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.980+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3380068610 con 0x7f33800cec00 2024-03-29T04:09:03.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.980+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f337c032a20 con 0x7f33800cec00 2024-03-29T04:09:03.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.981+0000 7f33767fc640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f336406c560 0x7f336406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:03.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.981+0000 7f3385bdd640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f336406c560 0x7f336406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:03.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.981+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f337c02f030 con 0x7f33800cec00 2024-03-29T04:09:03.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.981+0000 7f3385bdd640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f336406c560 0x7f336406ea20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3370000f60 tx=0x7f3370009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:03.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:03.985+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f337c041070 con 0x7f33800cec00 2024-03-29T04:09:04.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.204+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f33800b1d80 con 0x7f33800cec00 2024-03-29T04:09:04.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.205+0000 7f33767fc640 1 -- 172.21.15.156:0/647999257 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f337c07bdd0 con 0x7f33800cec00 2024-03-29T04:09:04.184 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:04.184 INFO:teuthology.orchestra.run.smithi156.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-29T04:09:04.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f336406c560 msgr2=0x7f336406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f336406c560 0x7f336406ea20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3370000f60 tx=0x7f3370009040 comp rx=0 tx=0).stop 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 msgr2=0x7f338014a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f338014a500 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f337c0097a0 tx=0x7f337c0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 shutdown_connections 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f336406c560 0x7f336406ea20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33801568f0 0x7f338014aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33800cec00 0x7f338014a500 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 --2- 172.21.15.156:0/647999257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33800cc870 0x7f33801383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 >> 172.21.15.156:0/647999257 conn(0x7f33800b3eb0 msgr2=0x7f33800b5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 shutdown_connections 2024-03-29T04:09:04.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.207+0000 7f3387e41640 1 -- 172.21.15.156:0/647999257 wait complete. 2024-03-29T04:09:04.197 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-29T04:09:04.198 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.318+0000 7f1884806640 1 Processor -- start 2024-03-29T04:09:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.318+0000 7f1884806640 1 -- start start 2024-03-29T04:09:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f1884806640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f188015b620 0x7f18800b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f1884806640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18800b05a0 0x7f18800ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f1884806640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18800afa60 con 0x7f188015b240 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f1884806640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f18800af4a0 con 0x7f18800b05a0 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f1884806640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f18800af780 con 0x7f188015b620 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f187ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18800b05a0 0x7f18800ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f187e59c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f188015b240 0x7f18800afe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47548/0 (socket says 172.21.15.156:47548) 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.319+0000 7f187e59c640 1 -- 172.21.15.156:0/2406461589 learned_addr learned my addr 172.21.15.156:0/2406461589 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187dd9b640 1 --2- 172.21.15.156:0/2406461589 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f188015b620 0x7f18800b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187ed9d640 1 -- 172.21.15.156:0/2406461589 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f188015b620 msgr2=0x7f18800b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187ed9d640 1 --2- 172.21.15.156:0/2406461589 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f188015b620 0x7f18800b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187d59a640 1 -- 172.21.15.156:0/2406461589 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358662791 0 0) 0x7f18800afa60 con 0x7f188015b240 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187ed9d640 1 -- 172.21.15.156:0/2406461589 >> v1:172.21.15.156:6789/0 conn(0x7f188015b240 legacy=0x7f18800afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187ed9d640 1 -- 172.21.15.156:0/2406461589 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18800b4100 con 0x7f18800b05a0 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.320+0000 7f187ed9d640 1 --2- 172.21.15.156:0/2406461589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18800b05a0 0x7f18800ae6b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f186800ba40 tx=0x7f186802e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7da7f93147476fb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f187d59a640 1 -- 172.21.15.156:0/2406461589 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186803b070 con 0x7f18800b05a0 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f187d59a640 1 -- 172.21.15.156:0/2406461589 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186802ee30 con 0x7f18800b05a0 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f187d59a640 1 -- 172.21.15.156:0/2406461589 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868036640 con 0x7f18800b05a0 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 -- 172.21.15.156:0/2406461589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18800b05a0 msgr2=0x7f18800ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 --2- 172.21.15.156:0/2406461589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18800b05a0 0x7f18800ae6b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f186800ba40 tx=0x7f186802e080 comp rx=0 tx=0).stop 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 -- 172.21.15.156:0/2406461589 shutdown_connections 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 --2- 172.21.15.156:0/2406461589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18800b05a0 0x7f18800ae6b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 --2- 172.21.15.156:0/2406461589 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f188015b620 0x7f18800b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 -- 172.21.15.156:0/2406461589 >> 172.21.15.156:0/2406461589 conn(0x7f188009aa40 msgr2=0x7f18800b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 -- 172.21.15.156:0/2406461589 shutdown_connections 2024-03-29T04:09:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.321+0000 7f1884806640 1 -- 172.21.15.156:0/2406461589 wait complete. 2024-03-29T04:09:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.322+0000 7f1884806640 1 Processor -- start 2024-03-29T04:09:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 -- start start 2024-03-29T04:09:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18800b05a0 0x7f188014fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f188015b240 0x7f18801f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 0x7f18801533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18800aed70 con 0x7f188015b620 2024-03-29T04:09:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f18800aebf0 con 0x7f188015b240 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f1884806640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f18800aeef0 con 0x7f18800b05a0 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f187ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 0x7f18801533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f187ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 0x7f18801533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48572/0 (socket says 172.21.15.156:48572) 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f187ed9d640 1 -- 172.21.15.156:0/398901948 learned_addr learned my addr 172.21.15.156:0/398901948 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f187dd9b640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f188015b240 0x7f18801f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.323+0000 7f187e59c640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18800b05a0 0x7f188014fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187ed9d640 1 -- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18800b05a0 msgr2=0x7f188014fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187ed9d640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18800b05a0 0x7f188014fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187ed9d640 1 -- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f188015b240 msgr2=0x7f18801f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187ed9d640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f188015b240 0x7f18801f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187ed9d640 1 -- 172.21.15.156:0/398901948 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1880152bc0 con 0x7f188015b620 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187dd9b640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f188015b240 0x7f18801f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f187ed9d640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 0x7f18801533a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f186800ba40 tx=0x7f18680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868045020 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1868040670 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18801f4270 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186803b040 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.324+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1880150260 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.325+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1880075a60 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.325+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1868048020 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.326+0000 7f1876ffd640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f186006c530 0x7f186006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.326+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f18680b0160 con 0x7f188015b620 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.326+0000 7f187e59c640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f186006c530 0x7f186006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.326+0000 7f187e59c640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f186006c530 0x7f186006e9f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18780012e0 tx=0x7f1878009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.310 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.330+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1880075a60 con 0x7f188015b620 2024-03-29T04:09:04.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.511+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1880068d20 con 0x7f188015b620 2024-03-29T04:09:04.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.512+0000 7f1876ffd640 1 -- 172.21.15.156:0/398901948 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f186804f030 con 0x7f188015b620 2024-03-29T04:09:04.492 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:04.492 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f186006c530 msgr2=0x7f186006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f186006c530 0x7f186006e9f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18780012e0 tx=0x7f1878009040 comp rx=0 tx=0).stop 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 msgr2=0x7f18801533a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 0x7f18801533a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f186800ba40 tx=0x7f18680046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 shutdown_connections 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f186006c530 0x7f186006e9f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f188015b620 0x7f18801533a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f188015b240 0x7f18801f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 --2- 172.21.15.156:0/398901948 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18800b05a0 0x7f188014fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 >> 172.21.15.156:0/398901948 conn(0x7f188009aa40 msgr2=0x7f18800b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:04.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 shutdown_connections 2024-03-29T04:09:04.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.515+0000 7f1884806640 1 -- 172.21.15.156:0/398901948 wait complete. 2024-03-29T04:09:04.505 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:04.506 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:04.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.630+0000 7f5d06715640 1 Processor -- start 2024-03-29T04:09:04.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.630+0000 7f5d06715640 1 -- start start 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5d06715640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5d000b44e0 0x7f5d000cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5d06715640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0015cb10 con 0x7f5d000cf340 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5d06715640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0015cdf0 con 0x7f5d00145c70 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5d06715640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5d0015d0d0 con 0x7f5d000b44e0 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5d04cb2640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5d000cf340 0x7f5d000d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47554/0 (socket says 172.21.15.156:47554) 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5d04cb2640 1 -- 172.21.15.156:0/521170778 learned_addr learned my addr 172.21.15.156:0/521170778 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/521170778 >> v2:172.21.15.203:3300/0 conn(0x7f5d000b44e0 0x7f5d000cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243350266 0 0) 0x7f5d0015cb10 con 0x7f5d000cf340 2024-03-29T04:09:04.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce8003520 con 0x7f5d000cf340 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2155608417 0 0) 0x7f5ce8003520 con 0x7f5d000cf340 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 >> v2:172.21.15.203:3300/0 conn(0x7f5d000b44e0 msgr2=0x7f5d000cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 --2- 172.21.15.156:0/521170778 >> v2:172.21.15.203:3300/0 conn(0x7f5d000b44e0 0x7f5d000cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 >> v1:172.21.15.203:6789/0 conn(0x7f5d00145c70 legacy=0x7f5d000b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0015d3b0 con 0x7f5d000cf340 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f5cf80019f0 con 0x7f5d000cf340 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/521170778 >> v2:172.21.15.203:3300/0 conn(0x7f5d000b44e0 0x7f5d000cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 >> v1:172.21.15.156:6789/0 conn(0x7f5d000cf340 legacy=0x7f5d000d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80053c0 0x7f5ce800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0015d0d0 con 0x7f5ce8004800 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce800c250 con 0x7f5ce80053c0 2024-03-29T04:09:04.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.632+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce800c450 con 0x7f5ce80039b0 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cfffff640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5d04cb2640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80053c0 0x7f5ce800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cff7fe640 1 -- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 msgr2=0x7f5ce8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cff7fe640 1 -- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80053c0 msgr2=0x7f5ce800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80053c0 0x7f5ce800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5d04cb2640 1 -- 172.21.15.156:0/521170778 reap_dead start 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.633+0000 7f5cff7fe640 1 -- 172.21.15.156:0/521170778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce8003520 con 0x7f5ce8004800 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5d0015a580 tx=0x7f5ce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58442c2de651b06f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf400f040 con 0x7f5ce8004800 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf4004030 con 0x7f5ce8004800 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5cfeffd640 1 -- 172.21.15.156:0/521170778 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4004440 con 0x7f5ce8004800 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 -- 172.21.15.156:0/521170778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 msgr2=0x7f5ce8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5d0015a580 tx=0x7f5ce8005830 comp rx=0 tx=0).stop 2024-03-29T04:09:04.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 -- 172.21.15.156:0/521170778 shutdown_connections 2024-03-29T04:09:04.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 --2- 172.21.15.156:0/521170778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 -- 172.21.15.156:0/521170778 >> 172.21.15.156:0/521170778 conn(0x7f5d000b7660 msgr2=0x7f5d000b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:04.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 -- 172.21.15.156:0/521170778 shutdown_connections 2024-03-29T04:09:04.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.634+0000 7f5d06715640 1 -- 172.21.15.156:0/521170778 wait complete. 2024-03-29T04:09:04.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.635+0000 7f5d06715640 1 Processor -- start 2024-03-29T04:09:04.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 -- start start 2024-03-29T04:09:04.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5d001449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d001f46c0 0x7f5d001e8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 0x7f5d0006d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001461d0 con 0x7f5d001e8660 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00146050 con 0x7f5d001f46c0 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d06715640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5d000cee00 con 0x7f5ce8004800 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d04cb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 0x7f5d0006d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d04cb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 0x7f5d0006d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48592/0 (socket says 172.21.15.156:48592) 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5d04cb2640 1 -- 172.21.15.156:0/1381206728 learned_addr learned my addr 172.21.15.156:0/1381206728 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.636+0000 7f5cff7fe640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d001f46c0 0x7f5d001e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5cfffff640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5d001449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d04cb2640 1 -- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 msgr2=0x7f5d001449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d04cb2640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5d001449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d04cb2640 1 -- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d001f46c0 msgr2=0x7f5d001e8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d04cb2640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d001f46c0 0x7f5d001e8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d04cb2640 1 -- 172.21.15.156:0/1381206728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d000cb810 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d04cb2640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 0x7f5d0006d2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5d0006ed30 tx=0x7f5cf8007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800aa00 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8019070 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d000cbaf0 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800ade0 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.637+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d000cc0b0 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.638+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d00160db0 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.638+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5cf801e460 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.639+0000 7f5cfcff9640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5cec06c530 0x7f5cec06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.639+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5cf808b810 con 0x7f5d001e8660 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.639+0000 7f5cfffff640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5cec06c530 0x7f5cec06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.639+0000 7f5cfffff640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5cec06c530 0x7f5cec06e9f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0001760 tx=0x7f5cf0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.643+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5d00160db0 con 0x7f5d001e8660 2024-03-29T04:09:04.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.824+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d00070e10 con 0x7f5d001e8660 2024-03-29T04:09:04.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.824+0000 7f5cfcff9640 1 -- 172.21.15.156:0/1381206728 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f5cf8059870 con 0x7f5d001e8660 2024-03-29T04:09:04.804 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5cec06c530 msgr2=0x7f5cec06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5cec06c530 0x7f5cec06e9f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0001760 tx=0x7f5cf0009040 comp rx=0 tx=0).stop 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 msgr2=0x7f5d0006d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 0x7f5d0006d2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5d0006ed30 tx=0x7f5cf8007a90 comp rx=0 tx=0).stop 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 shutdown_connections 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5cec06c530 0x7f5cec06e9f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d001e8660 0x7f5d0006d2e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d001f46c0 0x7f5d001e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 --2- 172.21.15.156:0/1381206728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5d001449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 >> 172.21.15.156:0/1381206728 conn(0x7f5d000b7660 msgr2=0x7f5d000b68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:04.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 shutdown_connections 2024-03-29T04:09:04.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.826+0000 7f5d06715640 1 -- 172.21.15.156:0/1381206728 wait complete. 2024-03-29T04:09:04.817 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:04.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.942+0000 7f7024f82640 1 Processor -- start 2024-03-29T04:09:04.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.942+0000 7f7024f82640 1 -- start start 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f7024f82640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f70200b44e0 0x7f70200cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f7024f82640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702015cb10 con 0x7f7020145c70 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f7024f82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702015cdf0 con 0x7f70200cf340 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f7024f82640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f702015d0d0 con 0x7f70200b44e0 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701e59c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7020145c70 0x7f70200b3dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47568/0 (socket says 172.21.15.156:47568) 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701e59c640 1 -- 172.21.15.156:0/1664797527 learned_addr learned my addr 172.21.15.156:0/1664797527 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:04.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f7017fff640 1 --2- 172.21.15.156:0/1664797527 >> v2:172.21.15.203:3300/0 conn(0x7f70200b44e0 0x7f70200cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700293560 0 0) 0x7f702015cb10 con 0x7f7020145c70 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f700c003520 con 0x7f7020145c70 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 167764200 0 0) 0x7f700c003520 con 0x7f7020145c70 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 >> v2:172.21.15.203:3300/0 conn(0x7f70200b44e0 msgr2=0x7f70200cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 --2- 172.21.15.156:0/1664797527 >> v2:172.21.15.203:3300/0 conn(0x7f70200b44e0 0x7f70200cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 >> v1:172.21.15.203:6789/0 conn(0x7f70200cf340 legacy=0x7f70200d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702015d3b0 con 0x7f7020145c70 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f7010001670 con 0x7f7020145c70 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 >> v1:172.21.15.156:6789/0 conn(0x7f7020145c70 legacy=0x7f70200b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f700c0039b0 0x7f700c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c004800 0x7f700c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f700c0052e0 0x7f700c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f702015d0d0 con 0x7f700c0052e0 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f700c00c1b0 con 0x7f700c004800 2024-03-29T04:09:04.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.943+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f700c00c3b0 con 0x7f700c0039b0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f700c0052e0 0x7f700c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701e59c640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f700c0039b0 0x7f700c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f7017fff640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c004800 0x7f700c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 -- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f700c0039b0 msgr2=0x7f700c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f700c0039b0 0x7f700c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 -- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c004800 msgr2=0x7f700c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c004800 0x7f700c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 -- 172.21.15.156:0/1664797527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700c003520 con 0x7f700c0052e0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.944+0000 7f701ed9d640 1 -- 172.21.15.156:0/1664797527 reap_dead start 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f701ed9d640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f700c0052e0 0x7f700c00b9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7018001e20 tx=0x7f700c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c372031bd9405f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018001a70 con 0x7f700c0052e0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7018003b70 con 0x7f700c0052e0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f701dd9b640 1 -- 172.21.15.156:0/1664797527 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70180050d0 con 0x7f700c0052e0 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 -- 172.21.15.156:0/1664797527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f700c0052e0 msgr2=0x7f700c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f700c0052e0 0x7f700c00b9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7018001e20 tx=0x7f700c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:04.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 -- 172.21.15.156:0/1664797527 shutdown_connections 2024-03-29T04:09:04.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 --2- 172.21.15.156:0/1664797527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f700c0052e0 0x7f700c00b9c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 -- 172.21.15.156:0/1664797527 >> 172.21.15.156:0/1664797527 conn(0x7f70200b7660 msgr2=0x7f70200b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:04.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 -- 172.21.15.156:0/1664797527 shutdown_connections 2024-03-29T04:09:04.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.945+0000 7f7024f82640 1 -- 172.21.15.156:0/1664797527 wait complete. 2024-03-29T04:09:04.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 Processor -- start 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 -- start start 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c0052e0 0x7f70201e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 0x7f702015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70201446c0 0x7f702013ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70201461d0 con 0x7f702013ef60 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7020146050 con 0x7f700c0052e0 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7024f82640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f70200cee00 con 0x7f70201446c0 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7017fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 0x7f702015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7017fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 0x7f702015ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48616/0 (socket says 172.21.15.156:48616) 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.947+0000 7f7017fff640 1 -- 172.21.15.156:0/525484089 learned_addr learned my addr 172.21.15.156:0/525484089 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f701e59c640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c0052e0 0x7f70201e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f701ed9d640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70201446c0 0x7f702013ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7017fff640 1 -- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70201446c0 msgr2=0x7f702013ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7017fff640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70201446c0 0x7f702013ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7017fff640 1 -- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c0052e0 msgr2=0x7f70201e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7017fff640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c0052e0 0x7f70201e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7017fff640 1 -- 172.21.15.156:0/525484089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70201e7420 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7017fff640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 0x7f702015ecc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7008011990 tx=0x7f7008011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008019070 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008004240 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700800f860 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70201e5e40 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.948+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70201e6400 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.949+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70200d0990 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.949+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f700801c070 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.950+0000 7f7016ffd640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f700006c530 0x7f700006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.950+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f700808baf0 con 0x7f702013ef60 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.950+0000 7f701e59c640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f700006c530 0x7f700006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:04.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.950+0000 7f701e59c640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f700006c530 0x7f700006e9f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f70100010e0 tx=0x7f7010009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:04.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:04.954+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f70200d0990 con 0x7f702013ef60 2024-03-29T04:09:05.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.133+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7020070e10 con 0x7f702013ef60 2024-03-29T04:09:05.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.133+0000 7f7016ffd640 1 -- 172.21.15.156:0/525484089 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f700805a120 con 0x7f702013ef60 2024-03-29T04:09:05.113 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:05.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f700006c530 msgr2=0x7f700006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f700006c530 0x7f700006e9f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f70100010e0 tx=0x7f7010009040 comp rx=0 tx=0).stop 2024-03-29T04:09:05.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 msgr2=0x7f702015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 0x7f702015ecc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7008011990 tx=0x7f7008011e50 comp rx=0 tx=0).stop 2024-03-29T04:09:05.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 shutdown_connections 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f700006c530 0x7f700006e9f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70201446c0 0x7f702013ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702013ef60 0x7f702015ecc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 --2- 172.21.15.156:0/525484089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f700c0052e0 0x7f70201e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 >> 172.21.15.156:0/525484089 conn(0x7f70200b7660 msgr2=0x7f702015a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 shutdown_connections 2024-03-29T04:09:05.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.135+0000 7f7024f82640 1 -- 172.21.15.156:0/525484089 wait complete. 2024-03-29T04:09:05.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:05.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.252+0000 7f58970a5640 1 Processor -- start 2024-03-29T04:09:05.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.252+0000 7f58970a5640 1 -- start start 2024-03-29T04:09:05.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f58970a5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5890071020 0x7f5890071400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f58970a5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5890141ca0 0x7f589015c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f58970a5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5890071940 con 0x7f589015ceb0 2024-03-29T04:09:05.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f58970a5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f589015d960 con 0x7f5890141ca0 2024-03-29T04:09:05.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f58970a5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f589013f7f0 con 0x7f5890071020 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f588ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5890141ca0 0x7f589015c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f5895642640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f589015ceb0 0x7f589015ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47576/0 (socket says 172.21.15.156:47576) 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f5895642640 1 -- 172.21.15.156:0/808250571 learned_addr learned my addr 172.21.15.156:0/808250571 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f5894e41640 1 --2- 172.21.15.156:0/808250571 >> v2:172.21.15.203:3300/0 conn(0x7f5890071020 0x7f5890071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f588ffff640 1 -- 172.21.15.156:0/808250571 >> v2:172.21.15.203:3300/0 conn(0x7f5890071020 msgr2=0x7f5890071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f588ffff640 1 --2- 172.21.15.156:0/808250571 >> v2:172.21.15.203:3300/0 conn(0x7f5890071020 0x7f5890071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f588ffff640 1 -- 172.21.15.156:0/808250571 >> v1:172.21.15.156:6789/0 conn(0x7f589015ceb0 legacy=0x7f589015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f588ffff640 1 -- 172.21.15.156:0/808250571 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589015d3a0 con 0x7f5890141ca0 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f588f7fe640 1 -- 172.21.15.156:0/808250571 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323379884 0 0) 0x7f5890071940 con 0x7f589015ceb0 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.253+0000 7f5894e41640 1 --2- 172.21.15.156:0/808250571 >> v2:172.21.15.203:3300/0 conn(0x7f5890071020 0x7f5890071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.254+0000 7f588ffff640 1 --2- 172.21.15.156:0/808250571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f589015c970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f589006c560 tx=0x7f587802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d63d7dd16552b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.254+0000 7f588f7fe640 1 -- 172.21.15.156:0/808250571 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878035020 con 0x7f5890141ca0 2024-03-29T04:09:05.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.254+0000 7f588f7fe640 1 -- 172.21.15.156:0/808250571 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878031830 con 0x7f5890141ca0 2024-03-29T04:09:05.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.254+0000 7f588f7fe640 1 -- 172.21.15.156:0/808250571 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878031c00 con 0x7f5890141ca0 2024-03-29T04:09:05.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.254+0000 7f58970a5640 1 -- 172.21.15.156:0/808250571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 msgr2=0x7f589015c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.254+0000 7f58970a5640 1 --2- 172.21.15.156:0/808250571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f589015c970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f589006c560 tx=0x7f587802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.255+0000 7f58970a5640 1 -- 172.21.15.156:0/808250571 shutdown_connections 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.255+0000 7f58970a5640 1 --2- 172.21.15.156:0/808250571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f589015c970 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.255+0000 7f58970a5640 1 --2- 172.21.15.156:0/808250571 >> v2:172.21.15.203:3300/0 conn(0x7f5890071020 0x7f5890071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.255+0000 7f58970a5640 1 -- 172.21.15.156:0/808250571 >> 172.21.15.156:0/808250571 conn(0x7f58900b3eb0 msgr2=0x7f58900b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.255+0000 7f58970a5640 1 -- 172.21.15.156:0/808250571 shutdown_connections 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.255+0000 7f58970a5640 1 -- 172.21.15.156:0/808250571 wait complete. 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.256+0000 7f58970a5640 1 Processor -- start 2024-03-29T04:09:05.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.256+0000 7f58970a5640 1 -- start start 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f58970a5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5890071020 0x7f58901385d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f58970a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f58900b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f58970a5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f589015ceb0 0x7f58900b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f58970a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58900d1280 con 0x7f5890141ca0 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f58970a5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f58900d0fa0 con 0x7f589015ceb0 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f58970a5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f58900d27b0 con 0x7f5890071020 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f58900b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f58900b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48640/0 (socket says 172.21.15.156:48640) 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 -- 172.21.15.156:0/1151695637 learned_addr learned my addr 172.21.15.156:0/1151695637 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 -- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5890071020 msgr2=0x7f58901385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f5895642640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f589015ceb0 0x7f58900b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f5894e41640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5890071020 0x7f58901385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5890071020 0x7f58901385d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 -- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f589015ceb0 msgr2=0x7f58900b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f589015ceb0 0x7f58900b3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.257+0000 7f588ffff640 1 -- 172.21.15.156:0/1151695637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589013c6c0 con 0x7f5890141ca0 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.258+0000 7f588ffff640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f58900b3310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f587802bee0 tx=0x7f58780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.258+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878041070 con 0x7f5890141ca0 2024-03-29T04:09:05.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.258+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f587803f070 con 0x7f5890141ca0 2024-03-29T04:09:05.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.258+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878035040 con 0x7f5890141ca0 2024-03-29T04:09:05.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.258+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5890138b10 con 0x7f5890141ca0 2024-03-29T04:09:05.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.258+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f589013be20 con 0x7f5890141ca0 2024-03-29T04:09:05.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.259+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5890070e10 con 0x7f5890141ca0 2024-03-29T04:09:05.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.260+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5878038020 con 0x7f5890141ca0 2024-03-29T04:09:05.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.260+0000 7f588d7fa640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f587006c530 0x7f587006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.260+0000 7f5894e41640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f587006c530 0x7f587006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.260+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f58780ad200 con 0x7f5890141ca0 2024-03-29T04:09:05.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.261+0000 7f5894e41640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f587006c530 0x7f587006e9f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5888009f70 tx=0x7f588800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.265+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f587807b790 con 0x7f5890141ca0 2024-03-29T04:09:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.444+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f589004f9d0 con 0x7f5890141ca0 2024-03-29T04:09:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.445+0000 7f588d7fa640 1 -- 172.21.15.156:0/1151695637 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f587807b090 con 0x7f5890141ca0 2024-03-29T04:09:05.425 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f587006c530 msgr2=0x7f587006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f587006c530 0x7f587006e9f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5888009f70 tx=0x7f588800b040 comp rx=0 tx=0).stop 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 msgr2=0x7f58900b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f58900b3310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f587802bee0 tx=0x7f58780046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 shutdown_connections 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f587006c530 0x7f587006e9f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f589015ceb0 0x7f58900b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5890141ca0 0x7f58900b3310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 --2- 172.21.15.156:0/1151695637 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5890071020 0x7f58901385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 >> 172.21.15.156:0/1151695637 conn(0x7f58900b3eb0 msgr2=0x7f589006f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 shutdown_connections 2024-03-29T04:09:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.447+0000 7f58970a5640 1 -- 172.21.15.156:0/1151695637 wait complete. 2024-03-29T04:09:05.437 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:05.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.559+0000 7f8bf678d640 1 Processor -- start 2024-03-29T04:09:05.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.559+0000 7f8bf678d640 1 -- start start 2024-03-29T04:09:05.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bf678d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8bf00b1850 0x7f8bf00b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bf678d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b32e0 0x7f8bf009aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bf678d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf00b1d00 con 0x7f8bf015fba0 2024-03-29T04:09:05.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bf678d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf00b1fe0 con 0x7f8bf00b1850 2024-03-29T04:09:05.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bf678d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf00b22c0 con 0x7f8bf00b32e0 2024-03-29T04:09:05.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bef7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8bf00b1850 0x7f8bf00b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bef7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8bf00b1850 0x7f8bf00b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48642/0 (socket says 172.21.15.156:48642) 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.560+0000 7f8bef7fe640 1 -- 172.21.15.156:0/1244419705 learned_addr learned my addr 172.21.15.156:0/1244419705 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8bf4d2a640 1 --2- 172.21.15.156:0/1244419705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b32e0 0x7f8bf009aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8bef7fe640 1 -- 172.21.15.156:0/1244419705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b32e0 msgr2=0x7f8bf009aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8bef7fe640 1 --2- 172.21.15.156:0/1244419705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b32e0 0x7f8bf009aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8bef7fe640 1 -- 172.21.15.156:0/1244419705 >> v1:172.21.15.203:6789/0 conn(0x7f8bf015fba0 legacy=0x7f8bf0160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8bef7fe640 1 -- 172.21.15.156:0/1244419705 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf00b25a0 con 0x7f8bf00b1850 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8bef7fe640 1 --2- 172.21.15.156:0/1244419705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf00b1850 0x7f8bf00b2da0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009870 tx=0x7f8be402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2291a95107c00b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8beeffd640 1 -- 172.21.15.156:0/1244419705 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40317c0 con 0x7f8bf00b1850 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.561+0000 7f8beeffd640 1 -- 172.21.15.156:0/1244419705 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be4037070 con 0x7f8bf00b1850 2024-03-29T04:09:05.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8beeffd640 1 -- 172.21.15.156:0/1244419705 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4031b80 con 0x7f8bf00b1850 2024-03-29T04:09:05.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 -- 172.21.15.156:0/1244419705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf00b1850 msgr2=0x7f8bf00b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 --2- 172.21.15.156:0/1244419705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf00b1850 0x7f8bf00b2da0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009870 tx=0x7f8be402b860 comp rx=0 tx=0).stop 2024-03-29T04:09:05.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 -- 172.21.15.156:0/1244419705 shutdown_connections 2024-03-29T04:09:05.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 --2- 172.21.15.156:0/1244419705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b32e0 0x7f8bf009aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 --2- 172.21.15.156:0/1244419705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf00b1850 0x7f8bf00b2da0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 -- 172.21.15.156:0/1244419705 >> 172.21.15.156:0/1244419705 conn(0x7f8bf00b3d90 msgr2=0x7f8bf00b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 -- 172.21.15.156:0/1244419705 shutdown_connections 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.562+0000 7f8bf678d640 1 -- 172.21.15.156:0/1244419705 wait complete. 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.563+0000 7f8bf678d640 1 Processor -- start 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.563+0000 7f8bf678d640 1 -- start start 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf678d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b1850 0x7f8bf00cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf678d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf00b32e0 0x7f8bf00c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf678d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 0x7f8bf00c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf678d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf009af80 con 0x7f8bf015fba0 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf678d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf00b3990 con 0x7f8bf00b32e0 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf678d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf009b100 con 0x7f8bf00b1850 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf4d2a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 0x7f8bf00c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf4d2a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 0x7f8bf00c6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48646/0 (socket says 172.21.15.156:48646) 2024-03-29T04:09:05.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bf4d2a640 1 -- 172.21.15.156:0/3931429800 learned_addr learned my addr 172.21.15.156:0/3931429800 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8bef7fe640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf00b32e0 0x7f8bf00c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.564+0000 7f8beffff640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b1850 0x7f8bf00cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf4d2a640 1 -- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b1850 msgr2=0x7f8bf00cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf4d2a640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b1850 0x7f8bf00cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf4d2a640 1 -- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf00b32e0 msgr2=0x7f8bf00c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf4d2a640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf00b32e0 0x7f8bf00c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf4d2a640 1 -- 172.21.15.156:0/3931429800 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf014b560 con 0x7f8bf015fba0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf4d2a640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 0x7f8bf00c6e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8be800cd70 tx=0x7f8be800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8003b90 con 0x7f8bf015fba0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be8003d30 con 0x7f8bf015fba0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf014b840 con 0x7f8bf015fba0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8004240 con 0x7f8bf015fba0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.565+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf014be00 con 0x7f8bf015fba0 2024-03-29T04:09:05.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.566+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf0068d20 con 0x7f8bf015fba0 2024-03-29T04:09:05.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.567+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8be80043e0 con 0x7f8bf015fba0 2024-03-29T04:09:05.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.567+0000 7f8becff9640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8bdc06c530 0x7f8bdc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.567+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8be808c270 con 0x7f8bf015fba0 2024-03-29T04:09:05.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.567+0000 7f8beffff640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8bdc06c530 0x7f8bdc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.567+0000 7f8beffff640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8bdc06c530 0x7f8bdc06e9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8be0006fd0 tx=0x7f8be0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.571+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8be801d340 con 0x7f8bf015fba0 2024-03-29T04:09:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.788+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bf0073bf0 con 0x7f8bf015fba0 2024-03-29T04:09:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.789+0000 7f8becff9640 1 -- 172.21.15.156:0/3931429800 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f8be805a810 con 0x7f8bf015fba0 2024-03-29T04:09:05.769 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:05.769 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8bdc06c530 msgr2=0x7f8bdc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8bdc06c530 0x7f8bdc06e9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8be0006fd0 tx=0x7f8be0008040 comp rx=0 tx=0).stop 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 msgr2=0x7f8bf00c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 0x7f8bf00c6e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8be800cd70 tx=0x7f8be800a660 comp rx=0 tx=0).stop 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 shutdown_connections 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8bdc06c530 0x7f8bdc06e9f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bf015fba0 0x7f8bf00c6e20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf00b32e0 0x7f8bf00c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 --2- 172.21.15.156:0/3931429800 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8bf00b1850 0x7f8bf00cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 >> 172.21.15.156:0/3931429800 conn(0x7f8bf00b3d90 msgr2=0x7f8bf00b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:05.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 shutdown_connections 2024-03-29T04:09:05.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.791+0000 7f8bf678d640 1 -- 172.21.15.156:0/3931429800 wait complete. 2024-03-29T04:09:05.782 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:05.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.906+0000 7f966be22640 1 Processor -- start 2024-03-29T04:09:05.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.906+0000 7f966be22640 1 -- start start 2024-03-29T04:09:05.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966be22640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96641568f0 0x7f96640cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966be22640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f96641609b0 0x7f966415d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966be22640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f966415cb50 con 0x7f96640cc870 2024-03-29T04:09:05.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966be22640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f96640cf8b0 con 0x7f96641609b0 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966be22640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9664160e30 con 0x7f96641568f0 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966a3bf640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f96640cc870 0x7f9664071020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47578/0 (socket says 172.21.15.156:47578) 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f966a3bf640 1 -- 172.21.15.156:0/4253978973 learned_addr learned my addr 172.21.15.156:0/4253978973 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9669bbe640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96641568f0 0x7f96640cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1490044357 0 0) 0x7f966415cb50 con 0x7f96640cc870 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f96693bd640 1 --2- 172.21.15.156:0/4253978973 >> v2:172.21.15.203:3300/0 conn(0x7f96641609b0 0x7f966415d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9650003520 con 0x7f96640cc870 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 783561911 0 0) 0x7f9650003520 con 0x7f96640cc870 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96641568f0 msgr2=0x7f96640cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96641568f0 0x7f96640cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 >> v2:172.21.15.203:3300/0 conn(0x7f96641609b0 msgr2=0x7f966415d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 --2- 172.21.15.156:0/4253978973 >> v2:172.21.15.203:3300/0 conn(0x7f96641609b0 0x7f966415d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96640cf2f0 con 0x7f96640cc870 2024-03-29T04:09:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f965c001a10 con 0x7f96640cc870 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 >> v1:172.21.15.156:6789/0 conn(0x7f96640cc870 legacy=0x7f9664071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96500039b0 0x7f9650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9650004800 0x7f9650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96500051c0 0x7f965000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f96640cf8b0 con 0x7f96500039b0 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9664160e30 con 0x7f96500051c0 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f965000c000 con 0x7f9650004800 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f96693bd640 1 --2- 172.21.15.156:0/4253978973 >> v2:172.21.15.203:3300/0 conn(0x7f96641609b0 0x7f966415d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.907+0000 7f9669bbe640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96641568f0 0x7f96640cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f9669bbe640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96500039b0 0x7f9650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f966a3bf640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96500051c0 0x7f965000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f96693bd640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9650004800 0x7f9650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f9669bbe640 1 -- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9650004800 msgr2=0x7f9650004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f9669bbe640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9650004800 0x7f9650004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f9669bbe640 1 -- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96500051c0 msgr2=0x7f965000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f9669bbe640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96500051c0 0x7f965000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.908+0000 7f9669bbe640 1 -- 172.21.15.156:0/4253978973 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9650003520 con 0x7f96500039b0 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.909+0000 7f9669bbe640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96500039b0 0x7f9650003dd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f96640d3280 tx=0x7f9650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ec9a72376eb707 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.909+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966000f040 con 0x7f96500039b0 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.909+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9660033600 con 0x7f96500039b0 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.909+0000 7f9668bbc640 1 -- 172.21.15.156:0/4253978973 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96600339d0 con 0x7f96500039b0 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966be22640 1 -- 172.21.15.156:0/4253978973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96500039b0 msgr2=0x7f9650003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966be22640 1 --2- 172.21.15.156:0/4253978973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96500039b0 0x7f9650003dd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f96640d3280 tx=0x7f9650005830 comp rx=0 tx=0).stop 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966a3bf640 1 -- 172.21.15.156:0/4253978973 reap_dead start 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966be22640 1 -- 172.21.15.156:0/4253978973 shutdown_connections 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966be22640 1 -- 172.21.15.156:0/4253978973 >> 172.21.15.156:0/4253978973 conn(0x7f96640b3eb0 msgr2=0x7f96640b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966be22640 1 -- 172.21.15.156:0/4253978973 shutdown_connections 2024-03-29T04:09:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.910+0000 7f966be22640 1 -- 172.21.15.156:0/4253978973 wait complete. 2024-03-29T04:09:05.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.911+0000 7f966be22640 1 Processor -- start 2024-03-29T04:09:05.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.911+0000 7f966be22640 1 -- start start 2024-03-29T04:09:05.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.912+0000 7f966be22640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96500039b0 0x7f9664138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.912+0000 7f966be22640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9664155c20 0x7f96641553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.912+0000 7f966be22640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 0x7f966413a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.912+0000 7f966be22640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f966415d850 con 0x7f9664139be0 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.912+0000 7f966be22640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f966415cdf0 con 0x7f9664155c20 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.912+0000 7f966be22640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9664156cd0 con 0x7f96500039b0 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 0x7f966413a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 0x7f966413a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48678/0 (socket says 172.21.15.156:48678) 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 -- 172.21.15.156:0/939089641 learned_addr learned my addr 172.21.15.156:0/939089641 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f96693bd640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9664155c20 0x7f96641553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f9669bbe640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96500039b0 0x7f9664138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 -- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96500039b0 msgr2=0x7f9664138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96500039b0 0x7f9664138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 -- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9664155c20 msgr2=0x7f96641553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9664155c20 0x7f96641553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 -- 172.21.15.156:0/939089641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966413a9e0 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966a3bf640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 0x7f966413a040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f965c004fd0 tx=0x7f965c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c00ac00 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f965c00adc0 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96640b5fd0 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c0133f0 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.913+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96640b6590 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.914+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9664068610 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.915+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f965c01b020 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.915+0000 7f965a7fc640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f964806c530 0x7f964806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.915+0000 7f9669bbe640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f964806c530 0x7f964806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.915+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f965c08b450 con 0x7f9664139be0 2024-03-29T04:09:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.916+0000 7f9669bbe640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f964806c530 0x7f964806e9f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9660036000 tx=0x7f9660031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:05.919+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f965c059420 con 0x7f9664139be0 2024-03-29T04:09:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.103+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f966415db60 con 0x7f9664139be0 2024-03-29T04:09:06.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.104+0000 7f965a7fc640 1 -- 172.21.15.156:0/939089641 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f965c05d720 con 0x7f9664139be0 2024-03-29T04:09:06.084 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:06.084 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:06.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f964806c530 msgr2=0x7f964806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f964806c530 0x7f964806e9f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9660036000 tx=0x7f9660031000 comp rx=0 tx=0).stop 2024-03-29T04:09:06.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 msgr2=0x7f966413a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 0x7f966413a040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f965c004fd0 tx=0x7f965c007a90 comp rx=0 tx=0).stop 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 shutdown_connections 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f964806c530 0x7f964806e9f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9664139be0 0x7f966413a040 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9664155c20 0x7f96641553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.106+0000 7f966be22640 1 --2- 172.21.15.156:0/939089641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96500039b0 0x7f9664138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.107+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 >> 172.21.15.156:0/939089641 conn(0x7f96640b3eb0 msgr2=0x7f966415e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.107+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 shutdown_connections 2024-03-29T04:09:06.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.107+0000 7f966be22640 1 -- 172.21.15.156:0/939089641 wait complete. 2024-03-29T04:09:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:06.098 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:06.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f496b7640 1 Processor -- start 2024-03-29T04:09:06.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f496b7640 1 -- start start 2024-03-29T04:09:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f496b7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6f440d2a00 0x7f6f440cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f496b7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f440d2fd0 con 0x7f6f440cfd60 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f496b7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f440d32b0 con 0x7f6f4415e540 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f496b7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6f4406c970 con 0x7f6f440d2a00 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f437fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f6f440cfd60 0x7f6f4406ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47592/0 (socket says 172.21.15.156:47592) 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.224+0000 7f6f437fe640 1 -- 172.21.15.156:0/1595199827 learned_addr learned my addr 172.21.15.156:0/1595199827 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f427fc640 1 --2- 172.21.15.156:0/1595199827 >> v2:172.21.15.203:3300/0 conn(0x7f6f440d2a00 0x7f6f440cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583879244 0 0) 0x7f6f440d2fd0 con 0x7f6f440cfd60 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f2c003520 con 0x7f6f440cfd60 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2766980015 0 0) 0x7f6f2c003520 con 0x7f6f440cfd60 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 >> v2:172.21.15.203:3300/0 conn(0x7f6f440d2a00 msgr2=0x7f6f440cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 --2- 172.21.15.156:0/1595199827 >> v2:172.21.15.203:3300/0 conn(0x7f6f440d2a00 0x7f6f440cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 >> v1:172.21.15.203:6789/0 conn(0x7f6f4415e540 legacy=0x7f6f440cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4406c600 con 0x7f6f440cfd60 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f6f3c0019f0 con 0x7f6f440cfd60 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 >> v1:172.21.15.156:6789/0 conn(0x7f6f440cfd60 legacy=0x7f6f4406ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f2c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f2c004800 0x7f6f2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f2c0051c0 0x7f6f2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4406c970 con 0x7f6f2c0039b0 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c00c090 con 0x7f6f2c004800 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2c014aa0 con 0x7f6f2c0051c0 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f2c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f427fc640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f2c004800 0x7f6f2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f437fe640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f2c0051c0 0x7f6f2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f42ffd640 1 -- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f2c0051c0 msgr2=0x7f6f2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f2c0051c0 0x7f6f2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f42ffd640 1 -- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f2c004800 msgr2=0x7f6f2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f2c004800 0x7f6f2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f42ffd640 1 -- 172.21.15.156:0/1595199827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c003520 con 0x7f6f2c0039b0 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.225+0000 7f6f437fe640 1 -- 172.21.15.156:0/1595199827 reap_dead start 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.226+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f2c003df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f34001750 tx=0x7f6f2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6770fcfab9d68c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.226+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34001370 con 0x7f6f2c0039b0 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.226+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f340052d0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.226+0000 7f6f41ffb640 1 -- 172.21.15.156:0/1595199827 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34003830 con 0x7f6f2c0039b0 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 -- 172.21.15.156:0/1595199827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 msgr2=0x7f6f2c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f2c003df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f34001750 tx=0x7f6f2c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 -- 172.21.15.156:0/1595199827 shutdown_connections 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 --2- 172.21.15.156:0/1595199827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f2c003df0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 -- 172.21.15.156:0/1595199827 >> 172.21.15.156:0/1595199827 conn(0x7f6f440d4450 msgr2=0x7f6f440d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 -- 172.21.15.156:0/1595199827 shutdown_connections 2024-03-29T04:09:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.227+0000 7f6f496b7640 1 -- 172.21.15.156:0/1595199827 wait complete. 2024-03-29T04:09:06.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.228+0000 7f6f496b7640 1 Processor -- start 2024-03-29T04:09:06.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.228+0000 7f6f496b7640 1 -- start start 2024-03-29T04:09:06.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f496b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f44157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f496b7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f44156550 0x7f6f440cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f496b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f44160e60 0x7f6f440b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f496b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f440cf110 con 0x7f6f2c0039b0 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f496b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6f440cf290 con 0x7f6f44160e60 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f496b7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6f440cf410 con 0x7f6f44156550 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f44157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f44157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48686/0 (socket says 172.21.15.156:48686) 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 -- 172.21.15.156:0/2623773202 learned_addr learned my addr 172.21.15.156:0/2623773202 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 -- 172.21.15.156:0/2623773202 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f44156550 msgr2=0x7f6f440cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f44156550 0x7f6f440cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 -- 172.21.15.156:0/2623773202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f44160e60 msgr2=0x7f6f440b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f44160e60 0x7f6f440b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 -- 172.21.15.156:0/2623773202 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f440cc5c0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.229+0000 7f6f42ffd640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f44157250 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6f34001870 tx=0x7f6f34001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.230+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34041020 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.230+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f34005480 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.230+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34007a30 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.230+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f44157830 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.230+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f44157df0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.231+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f440a23a0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.231+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6f34044020 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.232+0000 7f6f2b7fe640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f3006c560 0x7f6f3006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.232+0000 7f6f427fc640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f3006c560 0x7f6f3006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.232+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6f340ad2f0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.232+0000 7f6f427fc640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f3006c560 0x7f6f3006ea20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f38006e80 tx=0x7f6f38006df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.236+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6f3404b020 con 0x7f6f2c0039b0 2024-03-29T04:09:06.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.413+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6f4404f9d0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.414+0000 7f6f2b7fe640 1 -- 172.21.15.156:0/2623773202 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f6f340453b0 con 0x7f6f2c0039b0 2024-03-29T04:09:06.393 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:06.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f3006c560 msgr2=0x7f6f3006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f3006c560 0x7f6f3006ea20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f38006e80 tx=0x7f6f38006df0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 msgr2=0x7f6f44157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f44157250 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6f34001870 tx=0x7f6f34001120 comp rx=0 tx=0).stop 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 shutdown_connections 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f3006c560 0x7f6f3006ea20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f44160e60 0x7f6f440b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f44156550 0x7f6f440cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 --2- 172.21.15.156:0/2623773202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f2c0039b0 0x7f6f44157250 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 >> 172.21.15.156:0/2623773202 conn(0x7f6f440d4450 msgr2=0x7f6f440b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 shutdown_connections 2024-03-29T04:09:06.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.416+0000 7f6f496b7640 1 -- 172.21.15.156:0/2623773202 wait complete. 2024-03-29T04:09:06.406 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:06.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.530+0000 7fad6c8c5640 1 Processor -- start 2024-03-29T04:09:06.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.530+0000 7fad6c8c5640 1 -- start start 2024-03-29T04:09:06.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad6c8c5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fad6815e920 0x7fad680d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad6c8c5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fad680cfc30 0x7fad680d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad6c8c5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad680cf190 con 0x7fad680d2120 2024-03-29T04:09:06.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad6c8c5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fad680cf7a0 con 0x7fad6815e920 2024-03-29T04:09:06.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad6c8c5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fad680d3080 con 0x7fad680cfc30 2024-03-29T04:09:06.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad6659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fad6815e920 0x7fad680d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad65d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fad680d2120 0x7fad6806c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47596/0 (socket says 172.21.15.156:47596) 2024-03-29T04:09:06.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.531+0000 7fad65d9b640 1 -- 172.21.15.156:0/1462687205 learned_addr learned my addr 172.21.15.156:0/1462687205 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:06.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad66d9d640 1 --2- 172.21.15.156:0/1462687205 >> v2:172.21.15.203:3300/0 conn(0x7fad680cfc30 0x7fad680d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad6659c640 1 -- 172.21.15.156:0/1462687205 >> v2:172.21.15.203:3300/0 conn(0x7fad680cfc30 msgr2=0x7fad680d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:06.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad6659c640 1 --2- 172.21.15.156:0/1462687205 >> v2:172.21.15.203:3300/0 conn(0x7fad680cfc30 0x7fad680d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad6659c640 1 -- 172.21.15.156:0/1462687205 >> v1:172.21.15.156:6789/0 conn(0x7fad680d2120 legacy=0x7fad6806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad6559a640 1 -- 172.21.15.156:0/1462687205 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904369945 0 0) 0x7fad680cf190 con 0x7fad680d2120 2024-03-29T04:09:06.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad6659c640 1 -- 172.21.15.156:0/1462687205 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad680cf430 con 0x7fad6815e920 2024-03-29T04:09:06.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.532+0000 7fad66d9d640 1 --2- 172.21.15.156:0/1462687205 >> v2:172.21.15.203:3300/0 conn(0x7fad680cfc30 0x7fad680d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:06.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6659c640 1 --2- 172.21.15.156:0/1462687205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad680d26e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad6808ca00 tx=0x7fad6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf77bc3e3df619c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6559a640 1 -- 172.21.15.156:0/1462687205 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60035020 con 0x7fad6815e920 2024-03-29T04:09:06.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6559a640 1 -- 172.21.15.156:0/1462687205 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad60031830 con 0x7fad6815e920 2024-03-29T04:09:06.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6559a640 1 -- 172.21.15.156:0/1462687205 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60031c00 con 0x7fad6815e920 2024-03-29T04:09:06.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1462687205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 msgr2=0x7fad680d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1462687205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad680d26e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad6808ca00 tx=0x7fad6002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1462687205 shutdown_connections 2024-03-29T04:09:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1462687205 >> v2:172.21.15.203:3300/0 conn(0x7fad680cfc30 0x7fad680d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1462687205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad680d26e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1462687205 >> 172.21.15.156:0/1462687205 conn(0x7fad6809d7a0 msgr2=0x7fad680d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1462687205 shutdown_connections 2024-03-29T04:09:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.533+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1462687205 wait complete. 2024-03-29T04:09:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.535+0000 7fad6c8c5640 1 Processor -- start 2024-03-29T04:09:06.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.535+0000 7fad6c8c5640 1 -- start start 2024-03-29T04:09:06.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.536+0000 7fad6c8c5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad680cfc30 0x7fad680a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.536+0000 7fad6c8c5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fad680d2120 0x7fad6809e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6c8c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad681f8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6c8c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad680d34e0 con 0x7fad6815e920 2024-03-29T04:09:06.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6c8c5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad680d3360 con 0x7fad680cfc30 2024-03-29T04:09:06.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6c8c5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fad680d2c20 con 0x7fad680d2120 2024-03-29T04:09:06.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad681f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad681f8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48704/0 (socket says 172.21.15.156:48704) 2024-03-29T04:09:06.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 -- 172.21.15.156:0/1154834355 learned_addr learned my addr 172.21.15.156:0/1154834355 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:06.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad65d9b640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fad680d2120 0x7fad6809e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6659c640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad680cfc30 0x7fad680a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 -- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fad680d2120 msgr2=0x7fad6809e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fad680d2120 0x7fad6809e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 -- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad680cfc30 msgr2=0x7fad680a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad680cfc30 0x7fad680a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 -- 172.21.15.156:0/1154834355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6815b040 con 0x7fad6815e920 2024-03-29T04:09:06.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6659c640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad680cfc30 0x7fad680a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:06.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad66d9d640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad681f8190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad5800c7f0 tx=0x7fad5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58015070 con 0x7fad6815e920 2024-03-29T04:09:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad58007820 con 0x7fad6815e920 2024-03-29T04:09:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58007c50 con 0x7fad6815e920 2024-03-29T04:09:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad680a0c10 con 0x7fad6815e920 2024-03-29T04:09:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.537+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad680a11d0 con 0x7fad6815e920 2024-03-29T04:09:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.538+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad680b3690 con 0x7fad6815e920 2024-03-29T04:09:06.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.539+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fad58018070 con 0x7fad6815e920 2024-03-29T04:09:06.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.539+0000 7fad5effd640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fad4806c530 0x7fad4806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.539+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fad5808b870 con 0x7fad6815e920 2024-03-29T04:09:06.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.539+0000 7fad6659c640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fad4806c530 0x7fad4806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.539+0000 7fad6659c640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fad4806c530 0x7fad4806e9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fad60036000 tx=0x7fad6003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.548+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fad680b3690 con 0x7fad6815e920 2024-03-29T04:09:06.707 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.728+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fad6815def0 con 0x7fad6815e920 2024-03-29T04:09:06.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.728+0000 7fad5effd640 1 -- 172.21.15.156:0/1154834355 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fad58059e10 con 0x7fad6815e920 2024-03-29T04:09:06.708 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:06.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fad4806c530 msgr2=0x7fad4806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fad4806c530 0x7fad4806e9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fad60036000 tx=0x7fad6003c040 comp rx=0 tx=0).stop 2024-03-29T04:09:06.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 msgr2=0x7fad681f8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad681f8190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad5800c7f0 tx=0x7fad5800ccb0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 shutdown_connections 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fad4806c530 0x7fad4806e9f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad6815e920 0x7fad681f8190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fad680d2120 0x7fad6809e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 --2- 172.21.15.156:0/1154834355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad680cfc30 0x7fad680a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.730+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 >> 172.21.15.156:0/1154834355 conn(0x7fad6809d7a0 msgr2=0x7fad6806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.731+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 shutdown_connections 2024-03-29T04:09:06.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.731+0000 7fad6c8c5640 1 -- 172.21.15.156:0/1154834355 wait complete. 2024-03-29T04:09:06.721 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:06.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.845+0000 7ff426f82640 1 Processor -- start 2024-03-29T04:09:06.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.845+0000 7ff426f82640 1 -- start start 2024-03-29T04:09:06.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff426f82640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff4200d1f00 0x7ff4200cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff426f82640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42015cb10 con 0x7ff4200b44e0 2024-03-29T04:09:06.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff426f82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42015cdf0 con 0x7ff420145c70 2024-03-29T04:09:06.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff426f82640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff42015d0d0 con 0x7ff4200d1f00 2024-03-29T04:09:06.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff42551f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff4200d1f00 0x7ff4200cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff41ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff4200b44e0 0x7ff4200cee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47598/0 (socket says 172.21.15.156:47598) 2024-03-29T04:09:06.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.846+0000 7ff41ffff640 1 -- 172.21.15.156:0/1658288475 learned_addr learned my addr 172.21.15.156:0/1658288475 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:06.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff42551f640 1 -- 172.21.15.156:0/1658288475 >> v1:172.21.15.203:6789/0 conn(0x7ff420145c70 legacy=0x7ff4200b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff42551f640 1 -- 172.21.15.156:0/1658288475 >> v1:172.21.15.156:6789/0 conn(0x7ff4200b44e0 legacy=0x7ff4200cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff42551f640 1 -- 172.21.15.156:0/1658288475 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42015d3b0 con 0x7ff4200d1f00 2024-03-29T04:09:06.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff41f7fe640 1 -- 172.21.15.156:0/1658288475 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 414669396 0 0) 0x7ff42015cb10 con 0x7ff4200b44e0 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff42551f640 1 --2- 172.21.15.156:0/1658288475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200d1f00 0x7ff4200cf510 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff41802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a9facd6c7b950 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff41f7fe640 1 -- 172.21.15.156:0/1658288475 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418039070 con 0x7ff4200d1f00 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.847+0000 7ff41f7fe640 1 -- 172.21.15.156:0/1658288475 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff41802cbd0 con 0x7ff4200d1f00 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff41f7fe640 1 -- 172.21.15.156:0/1658288475 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418034440 con 0x7ff4200d1f00 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 -- 172.21.15.156:0/1658288475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200d1f00 msgr2=0x7ff4200cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 --2- 172.21.15.156:0/1658288475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200d1f00 0x7ff4200cf510 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff41802be20 comp rx=0 tx=0).stop 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 -- 172.21.15.156:0/1658288475 shutdown_connections 2024-03-29T04:09:06.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 --2- 172.21.15.156:0/1658288475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200d1f00 0x7ff4200cf510 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 -- 172.21.15.156:0/1658288475 >> 172.21.15.156:0/1658288475 conn(0x7ff4200b7660 msgr2=0x7ff4200b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:06.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 -- 172.21.15.156:0/1658288475 shutdown_connections 2024-03-29T04:09:06.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.848+0000 7ff426f82640 1 -- 172.21.15.156:0/1658288475 wait complete. 2024-03-29T04:09:06.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.849+0000 7ff426f82640 1 Processor -- start 2024-03-29T04:09:06.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.849+0000 7ff426f82640 1 -- start start 2024-03-29T04:09:06.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff426f82640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 0x7ff4200b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff426f82640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4200d1f00 0x7ff4200b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff426f82640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff420145c70 0x7ff42015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff426f82640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff420167490 con 0x7ff4200b44e0 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff426f82640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff420167310 con 0x7ff420145c70 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff426f82640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff420167610 con 0x7ff4200d1f00 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff424d1e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 0x7ff4200b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff424d1e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 0x7ff4200b5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48722/0 (socket says 172.21.15.156:48722) 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff424d1e640 1 -- 172.21.15.156:0/2857262796 learned_addr learned my addr 172.21.15.156:0/2857262796 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff41ffff640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4200d1f00 0x7ff4200b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.850+0000 7ff42551f640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff420145c70 0x7ff42015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.851+0000 7ff424d1e640 1 -- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4200d1f00 msgr2=0x7ff4200b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.851+0000 7ff424d1e640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4200d1f00 0x7ff4200b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.851+0000 7ff424d1e640 1 -- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff420145c70 msgr2=0x7ff42015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.851+0000 7ff424d1e640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff420145c70 0x7ff42015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:06.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.851+0000 7ff424d1e640 1 -- 172.21.15.156:0/2857262796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff420154020 con 0x7ff4200b44e0 2024-03-29T04:09:06.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.851+0000 7ff424d1e640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 0x7ff4200b5620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff41400c400 tx=0x7ff41400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.852+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414017070 con 0x7ff4200b44e0 2024-03-29T04:09:06.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.852+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4140052d0 con 0x7ff4200b44e0 2024-03-29T04:09:06.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.852+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4201f6190 con 0x7ff4200b44e0 2024-03-29T04:09:06.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.852+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414003830 con 0x7ff4200b44e0 2024-03-29T04:09:06.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.853+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4201f6750 con 0x7ff4200b44e0 2024-03-29T04:09:06.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.854+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff41401a070 con 0x7ff4200b44e0 2024-03-29T04:09:06.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.854+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff420070e10 con 0x7ff4200b44e0 2024-03-29T04:09:06.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.854+0000 7ff41d7fa640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff40406c560 0x7ff40406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:06.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.854+0000 7ff41ffff640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff40406c560 0x7ff40406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:06.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.854+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff41408a8c0 con 0x7ff4200b44e0 2024-03-29T04:09:06.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.855+0000 7ff41ffff640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff40406c560 0x7ff40406ea20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007f70 tx=0x7ff40c007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:06.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:06.859+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff414058920 con 0x7ff4200b44e0 2024-03-29T04:09:07.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.038+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff420159f40 con 0x7ff4200b44e0 2024-03-29T04:09:07.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.039+0000 7ff41d7fa640 1 -- 172.21.15.156:0/2857262796 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7ff41405cc20 con 0x7ff4200b44e0 2024-03-29T04:09:07.018 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:07.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.040+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff40406c560 msgr2=0x7ff40406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.040+0000 7ff426f82640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff40406c560 0x7ff40406ea20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007f70 tx=0x7ff40c007f00 comp rx=0 tx=0).stop 2024-03-29T04:09:07.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.040+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 msgr2=0x7ff4200b5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.040+0000 7ff426f82640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 0x7ff4200b5620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff41400c400 tx=0x7ff41400c8c0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 shutdown_connections 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff40406c560 0x7ff40406ea20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff420145c70 0x7ff42015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4200d1f00 0x7ff4200b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 --2- 172.21.15.156:0/2857262796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4200b44e0 0x7ff4200b5620 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 >> 172.21.15.156:0/2857262796 conn(0x7ff4200b7660 msgr2=0x7ff420141490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 shutdown_connections 2024-03-29T04:09:07.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.041+0000 7ff426f82640 1 -- 172.21.15.156:0/2857262796 wait complete. 2024-03-29T04:09:07.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:07.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.153+0000 7f9472cc8640 1 Processor -- start 2024-03-29T04:09:07.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.153+0000 7f9472cc8640 1 -- start start 2024-03-29T04:09:07.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.153+0000 7f9472cc8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f946c15e720 0x7f946c0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9472cc8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f946c0b40d0 0x7f946c06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9472cc8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c15eb00 con 0x7f946c06baf0 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9472cc8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f946c0b5b30 con 0x7f946c15e720 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9472cc8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f946c06bf40 con 0x7f946c0b40d0 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9470a64640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f946c15e720 0x7f946c0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9471265640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f946c06baf0 0x7f946c162240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47608/0 (socket says 172.21.15.156:47608) 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9471265640 1 -- 172.21.15.156:0/3874060342 learned_addr learned my addr 172.21.15.156:0/3874060342 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f946bfff640 1 --2- 172.21.15.156:0/3874060342 >> v2:172.21.15.203:3300/0 conn(0x7f946c0b40d0 0x7f946c06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9470a64640 1 -- 172.21.15.156:0/3874060342 >> v2:172.21.15.203:3300/0 conn(0x7f946c0b40d0 msgr2=0x7f946c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9470a64640 1 --2- 172.21.15.156:0/3874060342 >> v2:172.21.15.203:3300/0 conn(0x7f946c0b40d0 0x7f946c06b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9470a64640 1 -- 172.21.15.156:0/3874060342 >> v1:172.21.15.156:6789/0 conn(0x7f946c06baf0 legacy=0x7f946c162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f9470a64640 1 -- 172.21.15.156:0/3874060342 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c0b5d60 con 0x7f946c15e720 2024-03-29T04:09:07.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.154+0000 7f946b7fe640 1 -- 172.21.15.156:0/3874060342 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018877972 0 0) 0x7f946c15eb00 con 0x7f946c06baf0 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9470a64640 1 --2- 172.21.15.156:0/3874060342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c15e720 0x7f946c0b2560 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f946c09af10 tx=0x7f946002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e596b9de7f12c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f946b7fe640 1 -- 172.21.15.156:0/3874060342 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460035020 con 0x7f946c15e720 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f946b7fe640 1 -- 172.21.15.156:0/3874060342 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9460031830 con 0x7f946c15e720 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f946b7fe640 1 -- 172.21.15.156:0/3874060342 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460031c00 con 0x7f946c15e720 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 -- 172.21.15.156:0/3874060342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c15e720 msgr2=0x7f946c0b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 --2- 172.21.15.156:0/3874060342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c15e720 0x7f946c0b2560 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f946c09af10 tx=0x7f946002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 -- 172.21.15.156:0/3874060342 shutdown_connections 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 --2- 172.21.15.156:0/3874060342 >> v2:172.21.15.203:3300/0 conn(0x7f946c0b40d0 0x7f946c06b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 --2- 172.21.15.156:0/3874060342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c15e720 0x7f946c0b2560 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 -- 172.21.15.156:0/3874060342 >> 172.21.15.156:0/3874060342 conn(0x7f946c09aa40 msgr2=0x7f946c0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 -- 172.21.15.156:0/3874060342 shutdown_connections 2024-03-29T04:09:07.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.155+0000 7f9472cc8640 1 -- 172.21.15.156:0/3874060342 wait complete. 2024-03-29T04:09:07.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.157+0000 7f9472cc8640 1 Processor -- start 2024-03-29T04:09:07.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.157+0000 7f9472cc8640 1 -- start start 2024-03-29T04:09:07.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 0x7f946c161760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f946c0b40d0 0x7f946c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f946c15e720 0x7f946c141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0b6320 con 0x7f946c06baf0 2024-03-29T04:09:07.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0b2bb0 con 0x7f946c15e720 2024-03-29T04:09:07.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f946c0d1920 con 0x7f946c0b40d0 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 0x7f946c161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 0x7f946c161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48746/0 (socket says 172.21.15.156:48746) 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 -- 172.21.15.156:0/1402510078 learned_addr learned my addr 172.21.15.156:0/1402510078 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9471265640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f946c15e720 0x7f946c141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f946bfff640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f946c0b40d0 0x7f946c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 -- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f946c0b40d0 msgr2=0x7f946c155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f946c0b40d0 0x7f946c155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 -- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f946c15e720 msgr2=0x7f946c141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f946c15e720 0x7f946c141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 -- 172.21.15.156:0/1402510078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c155c00 con 0x7f946c06baf0 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9471265640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f946c15e720 0x7f946c141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:07.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9470a64640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 0x7f946c161760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9460000c00 tx=0x7f94600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460040070 con 0x7f946c06baf0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f946003a910 con 0x7f946c06baf0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460035040 con 0x7f946c06baf0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.159+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946c155ee0 con 0x7f946c06baf0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.160+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946c1564a0 con 0x7f946c06baf0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.161+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946c068d20 con 0x7f946c06baf0 2024-03-29T04:09:07.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.161+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f946004b060 con 0x7f946c06baf0 2024-03-29T04:09:07.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.161+0000 7f94697fa640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f945006c530 0x7f945006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.162+0000 7f946bfff640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f945006c530 0x7f945006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.162+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f94600aced0 con 0x7f946c06baf0 2024-03-29T04:09:07.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.162+0000 7f946bfff640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f945006c530 0x7f945006e9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b4580 tx=0x7f9458008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.166+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f946007b4f0 con 0x7f946c06baf0 2024-03-29T04:09:07.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.392+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f946c0aeb30 con 0x7f946c06baf0 2024-03-29T04:09:07.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.393+0000 7f94697fa640 1 -- 172.21.15.156:0/1402510078 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f9460031800 con 0x7f946c06baf0 2024-03-29T04:09:07.373 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:07.373 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:07.373 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f945006c530 msgr2=0x7f945006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f945006c530 0x7f945006e9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b4580 tx=0x7f9458008040 comp rx=0 tx=0).stop 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 msgr2=0x7f946c161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 0x7f946c161760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9460000c00 tx=0x7f94600046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 shutdown_connections 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f945006c530 0x7f945006e9f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f946c15e720 0x7f946c141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f946c0b40d0 0x7f946c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 --2- 172.21.15.156:0/1402510078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f946c06baf0 0x7f946c161760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 >> 172.21.15.156:0/1402510078 conn(0x7f946c09aa40 msgr2=0x7f946c0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 shutdown_connections 2024-03-29T04:09:07.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.395+0000 7f9472cc8640 1 -- 172.21.15.156:0/1402510078 wait complete. 2024-03-29T04:09:07.385 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:07.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.507+0000 7f6f8858d640 1 Processor -- start 2024-03-29T04:09:07.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.507+0000 7f6f8858d640 1 -- start start 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f8858d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f800b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f8858d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f800b1d00 con 0x7f6f8015fba0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f8858d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f800b1fe0 con 0x7f6f800b32e0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f8858d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6f800b22c0 con 0x7f6f800b1850 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f86329640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f6f8015fba0 0x7f6f80160a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47620/0 (socket says 172.21.15.156:47620) 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f86329640 1 -- 172.21.15.156:0/3516604298 learned_addr learned my addr 172.21.15.156:0/3516604298 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85b28640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f800b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592005985 0 0) 0x7f6f800b1d00 con 0x7f6f8015fba0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f6c003520 con 0x7f6f8015fba0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985900056 0 0) 0x7f6f800b1fe0 con 0x7f6f800b32e0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f800b1d00 con 0x7f6f800b32e0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3587079336 0 0) 0x7f6f6c003520 con 0x7f6f8015fba0 2024-03-29T04:09:07.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 msgr2=0x7f6f800b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f800b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 >> v1:172.21.15.203:6789/0 conn(0x7f6f800b32e0 legacy=0x7f6f8009aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f800b25a0 con 0x7f6f8015fba0 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f6f7c001670 con 0x7f6f8015fba0 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85b28640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f800b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 >> v1:172.21.15.156:6789/0 conn(0x7f6f8015fba0 legacy=0x7f6f80160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f6c003b70 0x7f6f6c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f800b22c0 con 0x7f6f6c003b70 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c00bad0 con 0x7f6f6c004800 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6f6c00bcd0 con 0x7f6f800b1850 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.508+0000 7f6f86329640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f6c003b70 0x7f6f6c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f85b28640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f86b2a640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f86329640 1 -- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 msgr2=0x7f6f6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f86329640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f800b1850 0x7f6f6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f86329640 1 -- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f6c004800 msgr2=0x7f6f6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f86329640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.509+0000 7f6f86329640 1 -- 172.21.15.156:0/3516604298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6c003520 con 0x7f6f6c003b70 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f86329640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f6c003b70 0x7f6f6c003f70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c007fd0 tx=0x7f6f6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d685f47b0ed70917 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c007ce0 con 0x7f6f6c003b70 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7c030070 con 0x7f6f6c003b70 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f85327640 1 -- 172.21.15.156:0/3516604298 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c005d10 con 0x7f6f6c003b70 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f8858d640 1 -- 172.21.15.156:0/3516604298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f6c003b70 msgr2=0x7f6f6c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3516604298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f6c003b70 0x7f6f6c003f70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c007fd0 tx=0x7f6f6c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f86b2a640 1 -- 172.21.15.156:0/3516604298 reap_dead start 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f8858d640 1 -- 172.21.15.156:0/3516604298 shutdown_connections 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f8858d640 1 -- 172.21.15.156:0/3516604298 >> 172.21.15.156:0/3516604298 conn(0x7f6f800b3d90 msgr2=0x7f6f800b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:07.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f8858d640 1 -- 172.21.15.156:0/3516604298 shutdown_connections 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.510+0000 7f6f8858d640 1 -- 172.21.15.156:0/3516604298 wait complete. 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.512+0000 7f6f8858d640 1 Processor -- start 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.512+0000 7f6f8858d640 1 -- start start 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f8858d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f6c003b70 0x7f6f80155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f8858d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 0x7f6f800cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f8858d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f8015d930 0x7f6f8015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f8858d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8009b150 con 0x7f6f800cb540 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f8858d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6f800b3990 con 0x7f6f8015d930 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f8858d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8009b2d0 con 0x7f6f6c003b70 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 0x7f6f800cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 0x7f6f800cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48776/0 (socket says 172.21.15.156:48776) 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 -- 172.21.15.156:0/3436633880 learned_addr learned my addr 172.21.15.156:0/3436633880 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f86329640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f6c003b70 0x7f6f80155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f86b2a640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f8015d930 0x7f6f8015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 -- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f6c003b70 msgr2=0x7f6f80155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f6c003b70 0x7f6f80155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 -- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f8015d930 msgr2=0x7f6f8015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f8015d930 0x7f6f8015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 -- 172.21.15.156:0/3436633880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8015e2f0 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f86329640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f6c003b70 0x7f6f80155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f86b2a640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f8015d930 0x7f6f8015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.513+0000 7f6f85b28640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 0x7f6f800cb9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f70011990 tx=0x7f6f70011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.514+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000fa70 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.514+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7000fc10 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.514+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8015b810 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.514+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70004030 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.514+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8015bdd0 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.515+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f80068d20 con 0x7f6f800cb540 2024-03-29T04:09:07.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.515+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6f70012040 con 0x7f6f800cb540 2024-03-29T04:09:07.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.516+0000 7f6f76ffd640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f6406c560 0x7f6f6406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.516+0000 7f6f86329640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f6406c560 0x7f6f6406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.516+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6f7008ba70 con 0x7f6f800cb540 2024-03-29T04:09:07.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.516+0000 7f6f86329640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f6406c560 0x7f6f6406ea20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c034000 tx=0x7f6f7c039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.520+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6f70019070 con 0x7f6f800cb540 2024-03-29T04:09:07.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.703+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f80073bf0 con 0x7f6f800cb540 2024-03-29T04:09:07.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.705+0000 7f6f76ffd640 1 -- 172.21.15.156:0/3436633880 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f6f7005a090 con 0x7f6f800cb540 2024-03-29T04:09:07.684 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:07.684 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f6406c560 msgr2=0x7f6f6406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f6406c560 0x7f6f6406ea20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c034000 tx=0x7f6f7c039040 comp rx=0 tx=0).stop 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 msgr2=0x7f6f800cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 0x7f6f800cb9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f70011990 tx=0x7f6f70011e50 comp rx=0 tx=0).stop 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 shutdown_connections 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6f6406c560 0x7f6f6406ea20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f8015d930 0x7f6f8015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f800cb540 0x7f6f800cb9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 --2- 172.21.15.156:0/3436633880 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f6c003b70 0x7f6f80155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 >> 172.21.15.156:0/3436633880 conn(0x7f6f800b3d90 msgr2=0x7f6f800b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 shutdown_connections 2024-03-29T04:09:07.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.707+0000 7f6f8858d640 1 -- 172.21.15.156:0/3436633880 wait complete. 2024-03-29T04:09:07.698 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:07.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.823+0000 7f50dc515640 1 Processor -- start 2024-03-29T04:09:07.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.823+0000 7f50dc515640 1 -- start start 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50dc515640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f50d4162340 0x7f50d40b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50dc515640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d4162e30 0x7f50d40d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50dc515640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d415fcf0 con 0x7f50d40b1bb0 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50dc515640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f50d4163280 con 0x7f50d4162340 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50dc515640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f50d40b2040 con 0x7f50d4162e30 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50da2b1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f50d4162340 0x7f50d40b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50d9ab0640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f50d40b1bb0 0x7f50d4162720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47636/0 (socket says 172.21.15.156:47636) 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50d9ab0640 1 -- 172.21.15.156:0/1497019166 learned_addr learned my addr 172.21.15.156:0/1497019166 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50daab2640 1 --2- 172.21.15.156:0/1497019166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d4162e30 0x7f50d40d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50da2b1640 1 -- 172.21.15.156:0/1497019166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d4162e30 msgr2=0x7f50d40d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50da2b1640 1 --2- 172.21.15.156:0/1497019166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d4162e30 0x7f50d40d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50da2b1640 1 -- 172.21.15.156:0/1497019166 >> v1:172.21.15.156:6789/0 conn(0x7f50d40b1bb0 legacy=0x7f50d4162720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50da2b1640 1 -- 172.21.15.156:0/1497019166 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d40cdc10 con 0x7f50d4162340 2024-03-29T04:09:07.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.824+0000 7f50d92af640 1 -- 172.21.15.156:0/1497019166 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1294754484 0 0) 0x7f50d415fcf0 con 0x7f50d40b1bb0 2024-03-29T04:09:07.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.825+0000 7f50da2b1640 1 --2- 172.21.15.156:0/1497019166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40b31e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50d00097a0 tx=0x7f50d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668eccdfb81f488c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.825+0000 7f50d92af640 1 -- 172.21.15.156:0/1497019166 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0035020 con 0x7f50d4162340 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.825+0000 7f50d92af640 1 -- 172.21.15.156:0/1497019166 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d0031830 con 0x7f50d4162340 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.825+0000 7f50d92af640 1 -- 172.21.15.156:0/1497019166 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0031c00 con 0x7f50d4162340 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 -- 172.21.15.156:0/1497019166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 msgr2=0x7f50d40b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 --2- 172.21.15.156:0/1497019166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40b31e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50d00097a0 tx=0x7f50d002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 -- 172.21.15.156:0/1497019166 shutdown_connections 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 --2- 172.21.15.156:0/1497019166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d4162e30 0x7f50d40d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 --2- 172.21.15.156:0/1497019166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40b31e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 -- 172.21.15.156:0/1497019166 >> 172.21.15.156:0/1497019166 conn(0x7f50d40b3d90 msgr2=0x7f50d40b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 -- 172.21.15.156:0/1497019166 shutdown_connections 2024-03-29T04:09:07.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.826+0000 7f50dc515640 1 -- 172.21.15.156:0/1497019166 wait complete. 2024-03-29T04:09:07.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.828+0000 7f50dc515640 1 Processor -- start 2024-03-29T04:09:07.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.828+0000 7f50dc515640 1 -- start start 2024-03-29T04:09:07.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50dc515640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d40b1bb0 0x7f50d41f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50dc515640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50dc515640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4162e30 0x7f50d40b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50dc515640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40cff10 con 0x7f50d4162340 2024-03-29T04:09:07.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50dc515640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40cd9c0 con 0x7f50d4162e30 2024-03-29T04:09:07.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50dc515640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f50d40d01f0 con 0x7f50d40b1bb0 2024-03-29T04:09:07.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50d9ab0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50d9ab0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48786/0 (socket says 172.21.15.156:48786) 2024-03-29T04:09:07.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50d9ab0640 1 -- 172.21.15.156:0/4226962244 learned_addr learned my addr 172.21.15.156:0/4226962244 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:07.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50da2b1640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d40b1bb0 0x7f50d41f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.829+0000 7f50daab2640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4162e30 0x7f50d40b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50d9ab0640 1 -- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d40b1bb0 msgr2=0x7f50d41f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50d9ab0640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d40b1bb0 0x7f50d41f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50d9ab0640 1 -- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4162e30 msgr2=0x7f50d40b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:07.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50d9ab0640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4162e30 0x7f50d40b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:07.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50d9ab0640 1 -- 172.21.15.156:0/4226962244 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d40c6240 con 0x7f50d4162340 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50daab2640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4162e30 0x7f50d40b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50d9ab0640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40c5d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50c400c500 tx=0x7f50c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c4018070 con 0x7f50d4162340 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c4007580 con 0x7f50d4162340 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d40cbd80 con 0x7f50d4162340 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c4004a20 con 0x7f50d4162340 2024-03-29T04:09:07.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.830+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d406c400 con 0x7f50d4162340 2024-03-29T04:09:07.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.831+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d4068d20 con 0x7f50d4162340 2024-03-29T04:09:07.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.831+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f50c401b070 con 0x7f50d4162340 2024-03-29T04:09:07.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.832+0000 7f50caffd640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f50b806c530 0x7f50b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:07.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.832+0000 7f50da2b1640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f50b806c530 0x7f50b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:07.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.832+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f50c408a9a0 con 0x7f50d4162340 2024-03-29T04:09:07.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.832+0000 7f50da2b1640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f50b806c530 0x7f50b806e9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f50d0036000 tx=0x7f50d0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:07.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:07.836+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f50c4059050 con 0x7f50d4162340 2024-03-29T04:09:08.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.111+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50d40b4940 con 0x7f50d4162340 2024-03-29T04:09:08.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.112+0000 7f50caffd640 1 -- 172.21.15.156:0/4226962244 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f50c4058950 con 0x7f50d4162340 2024-03-29T04:09:08.092 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:08.092 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:08.092 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f50b806c530 msgr2=0x7f50b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f50b806c530 0x7f50b806e9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f50d0036000 tx=0x7f50d0008030 comp rx=0 tx=0).stop 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 msgr2=0x7f50d40c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40c5d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50c400c500 tx=0x7f50c400c9c0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 shutdown_connections 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f50b806c530 0x7f50b806e9f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4162e30 0x7f50d40b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50d4162340 0x7f50d40c5d00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 --2- 172.21.15.156:0/4226962244 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50d40b1bb0 0x7f50d41f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 >> 172.21.15.156:0/4226962244 conn(0x7f50d40b3d90 msgr2=0x7f50d415a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 shutdown_connections 2024-03-29T04:09:08.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.114+0000 7f50dc515640 1 -- 172.21.15.156:0/4226962244 wait complete. 2024-03-29T04:09:08.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T04:08:53.151747+0000', 'modified': '2024-03-29T04:08:56.192682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6837/2168836037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 2168836037}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 2168836037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4290, 'qdb_cluster': [4290]} 2024-03-29T04:09:08.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T04:09:08.508 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-29T04:09:08.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.632+0000 7fbe5f352640 1 Processor -- start 2024-03-29T04:09:08.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.632+0000 7fbe5f352640 1 -- start start 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5f352640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 0x7fbe580d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5f352640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbe580d2120 0x7fbe5806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5f352640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe580cf190 con 0x7fbe5806cb10 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5f352640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbe580cf7a0 con 0x7fbe580d2120 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5f352640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbe580d3080 con 0x7fbe5815e920 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5d8ef640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbe5806cb10 0x7fbe580cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47646/0 (socket says 172.21.15.156:47646) 2024-03-29T04:09:08.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/3402767728 learned_addr learned my addr 172.21.15.156:0/3402767728 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403229762 0 0) 0x7fbe580cf190 con 0x7fbe5806cb10 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe44003520 con 0x7fbe5806cb10 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/3402767728 >> v2:172.21.15.203:3300/0 conn(0x7fbe580d2120 0x7fbe5806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 0x7fbe580d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 229845870 0 0) 0x7fbe44003520 con 0x7fbe5806cb10 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 msgr2=0x7fbe580d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 0x7fbe580d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 >> v2:172.21.15.203:3300/0 conn(0x7fbe580d2120 msgr2=0x7fbe5806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 --2- 172.21.15.156:0/3402767728 >> v2:172.21.15.203:3300/0 conn(0x7fbe580d2120 0x7fbe5806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe580cf430 con 0x7fbe5806cb10 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fbe500019f0 con 0x7fbe5806cb10 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 >> v1:172.21.15.156:6789/0 conn(0x7fbe5806cb10 legacy=0x7fbe580cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe440039b0 0x7fbe44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe44004800 0x7fbe44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 0x7fbe440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe580cf7a0 con 0x7fbe440039b0 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe580d3080 con 0x7fbe44004800 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.633+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbe4400b9c0 con 0x7fbe5815e920 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe440039b0 0x7fbe44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe44004800 0x7fbe44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 0x7fbe440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 msgr2=0x7fbe440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe5815e920 0x7fbe440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe44004800 msgr2=0x7fbe44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe44004800 0x7fbe44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3402767728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe44003520 con 0x7fbe440039b0 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.634+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe440039b0 0x7fbe44003dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009ac0 tx=0x7fbe44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e001e295b8ce94 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54039070 con 0x7fbe440039b0 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5402c6b0 con 0x7fbe440039b0 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe4ffff640 1 -- 172.21.15.156:0/3402767728 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54009690 con 0x7fbe440039b0 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5f352640 1 -- 172.21.15.156:0/3402767728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe440039b0 msgr2=0x7fbe44003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5f352640 1 --2- 172.21.15.156:0/3402767728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe440039b0 0x7fbe44003dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009ac0 tx=0x7fbe44005830 comp rx=0 tx=0).stop 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/3402767728 reap_dead start 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5f352640 1 -- 172.21.15.156:0/3402767728 shutdown_connections 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5f352640 1 -- 172.21.15.156:0/3402767728 >> 172.21.15.156:0/3402767728 conn(0x7fbe5809d7a0 msgr2=0x7fbe580d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5f352640 1 -- 172.21.15.156:0/3402767728 shutdown_connections 2024-03-29T04:09:08.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.635+0000 7fbe5f352640 1 -- 172.21.15.156:0/3402767728 wait complete. 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.636+0000 7fbe5f352640 1 Processor -- start 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.636+0000 7fbe5f352640 1 -- start start 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5f352640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe440039b0 0x7fbe5814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5f352640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe580b3e70 0x7fbe580ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5f352640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 0x7fbe5814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5f352640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe580d04c0 con 0x7fbe580ca8f0 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5f352640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe580d0340 con 0x7fbe580b3e70 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5f352640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbe580d26e0 con 0x7fbe440039b0 2024-03-29T04:09:08.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5d8ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 0x7fbe5814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5d8ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 0x7fbe5814d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48810/0 (socket says 172.21.15.156:48810) 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/865516067 learned_addr learned my addr 172.21.15.156:0/865516067 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe580b3e70 0x7fbe580ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe440039b0 0x7fbe5814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe440039b0 msgr2=0x7fbe5814c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.637+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe440039b0 0x7fbe5814c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe580b3e70 msgr2=0x7fbe580ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe580b3e70 0x7fbe580ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/865516067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe580c9310 con 0x7fbe580ca8f0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe440039b0 0x7fbe5814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 0x7fbe5814d090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbe500013d0 tx=0x7fbe5000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe4dffb640 1 -- 172.21.15.156:0/865516067 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5000ee20 con 0x7fbe580ca8f0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe4dffb640 1 -- 172.21.15.156:0/865516067 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe50014430 con 0x7fbe580ca8f0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe580c95f0 con 0x7fbe580ca8f0 2024-03-29T04:09:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe4dffb640 1 -- 172.21.15.156:0/865516067 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50014820 con 0x7fbe580ca8f0 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.638+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe580c9bb0 con 0x7fbe580ca8f0 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.639+0000 7fbe4dffb640 1 -- 172.21.15.156:0/865516067 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbe500149c0 con 0x7fbe580ca8f0 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.640+0000 7fbe4dffb640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbe3c06c530 0x7fbe3c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.640+0000 7fbe4dffb640 1 -- 172.21.15.156:0/865516067 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe5008b600 con 0x7fbe580ca8f0 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.640+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbe3c06c530 0x7fbe3c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.640+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbe3c06c530 0x7fbe3c06e9f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbe54036000 tx=0x7fbe54030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.641+0000 7fbe6201a740 1 Processor -- start 2024-03-29T04:09:08.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.641+0000 7fbe6201a740 1 -- start start 2024-03-29T04:09:08.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.642+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3a0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3a0d40 0x555d7a3d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.625 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe6201a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555d7a30a020 con 0x555d7a3cd860 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d7a2b25f0 con 0x555d7a3cdc40 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x555d7a305fd0 con 0x555d7a3a0d40 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48812/0 (socket says 172.21.15.156:48812) 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3052463501 learned_addr learned my addr 172.21.15.156:0/3052463501 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3052463501 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3a0d40 msgr2=0x555d7a3d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3a0d40 0x555d7a3d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3052463501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 msgr2=0x555d7a3d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/3052463501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d7a01e2d0 con 0x555d7a3cd860 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3a0800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe54037040 tx=0x7fbe5402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1e800e3c5be1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe42ffd640 1 -- 172.21.15.156:0/3052463501 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54039040 con 0x555d7a3cd860 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe42ffd640 1 -- 172.21.15.156:0/3052463501 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe540044c0 con 0x555d7a3cd860 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.643+0000 7fbe42ffd640 1 -- 172.21.15.156:0/3052463501 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5402cd80 con 0x555d7a3cd860 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 -- 172.21.15.156:0/3052463501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 msgr2=0x555d7a3a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3a0800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe54037040 tx=0x7fbe5402e000 comp rx=0 tx=0).stop 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 -- 172.21.15.156:0/3052463501 shutdown_connections 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3a0d40 0x555d7a3d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 --2- 172.21.15.156:0/3052463501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3a0800 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 -- 172.21.15.156:0/3052463501 >> 172.21.15.156:0/3052463501 conn(0x555d7a3a2a70 msgr2=0x555d7a3eb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 -- 172.21.15.156:0/3052463501 shutdown_connections 2024-03-29T04:09:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.644+0000 7fbe6201a740 1 -- 172.21.15.156:0/3052463501 wait complete. 2024-03-29T04:09:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.645+0000 7fbe6201a740 1 -- start start 2024-03-29T04:09:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.645+0000 7fbe6201a740 1 Processor -- start 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.646+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3eab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.646+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.646+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3d29d0 0x555d7a3fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.646+0000 7fbe6201a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555d7a3a5110 con 0x555d7a3cd860 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.646+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d7a2c59c0 con 0x555d7a3cdc40 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.646+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x555d7a3a7950 con 0x555d7a3d29d0 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe6201a740 10 client.0 _ll_register_callbacks cb 0x555d7a2a9fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3eab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3eab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48822/0 (socket says 172.21.15.156:48822) 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/1300582471 learned_addr learned my addr 172.21.15.156:0/1300582471 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3d29d0 0x555d7a3fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3d29d0 msgr2=0x555d7a3fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3d29d0 0x555d7a3fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 msgr2=0x555d7a3e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 -- 172.21.15.156:0/1300582471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d7a0e86b0 con 0x555d7a3cd860 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3d29d0 0x555d7a3fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3eab50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbe54037040 tx=0x7fbe540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca3a0a22282131a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe37fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe37fff640 1 -- 172.21.15.156:0/1300582471 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54039040 con 0x555d7a3cd860 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe37fff640 1 -- 172.21.15.156:0/1300582471 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe54048040 con 0x555d7a3cd860 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d7a2de4e0 con 0x555d7a3cd860 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe37fff640 1 -- 172.21.15.156:0/1300582471 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fbe5405e050 con 0x555d7a3cd860 2024-03-29T04:09:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe6201a740 10 client.4695 fetch_fsmap learned FSMap version 7 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x555d7a3e3350 con 0x555d7a3cd860 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.647+0000 7fbe37fff640 1 -- 172.21.15.156:0/1300582471 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x555d7a2de4e0 con 0x555d7a3cd860 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe37fff640 1 -- 172.21.15.156:0/1300582471 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7fbe54061020 con 0x555d7a3cd860 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe6201a740 10 client.4695 fetch_fsmap finished waiting for FSMap version 7 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe6201a740 10 client.4695 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe6201a740 20 client.4695 populate_metadata read hostname 'smithi156' 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe6201a740 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a3d2e30 0x555d7a3d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe6201a740 4 client.4695 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 --> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d7a317560 con 0x555d7a3d2e30 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a3d2e30 0x555d7a3d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe37fff640 10 client.4695 ms_handle_connect on v2:172.21.15.156:6836/2168836037 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.648+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a3d2e30 0x555d7a3d1790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=766107cbdf0b66a4 server_cookie=ebf71bc5a8a7dbc4 in_seq=0 out_seq=0 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe37fff640 1 -- 172.21.15.156:0/1300582471 <== mds.0 v2:172.21.15.156:6836/2168836037 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23239 (crc 0 0 0) 0x555d7a317560 con 0x555d7a3d2e30 2024-03-29T04:09:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe37fff640 10 client.4695 handle_command_reply: tid=0 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 client.4695 shutdown 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 msgr2=0x555d7a3eab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3eab50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbe54037040 tx=0x7fbe540046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 shutdown_connections 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3d29d0 0x555d7a3fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a3d2e30 0x555d7a3d1790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 --2- 172.21.15.156:0/1300582471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3cd860 0x555d7a3eab50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 >> 172.21.15.156:0/1300582471 conn(0x555d7a3a0d40 msgr2=0x555d7a3d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe37fff640 0 client.4695 ms_handle_reset on v2:172.21.15.156:6836/2168836037 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 shutdown_connections 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.649+0000 7fbe6201a740 1 -- 172.21.15.156:0/1300582471 wait complete. 2024-03-29T04:09:08.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.650+0000 7fbe6201a740 20 client.4695 trim_cache size 0 max 16384 2024-03-29T04:09:08.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.668+0000 7fbe6201a740 1 Processor -- start 2024-03-29T04:09:08.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.668+0000 7fbe6201a740 1 -- start start 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cd860 0x555d7a3d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a3d3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe6201a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe582001e0 con 0x555d7a3d29d0 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe580c8b70 con 0x555d7a3cdc40 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbe58160470 con 0x555d7a3cd860 2024-03-29T04:09:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a3d3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a3d3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48828/0 (socket says 172.21.15.156:48828) 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/613240032 learned_addr learned my addr 172.21.15.156:0/613240032 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cd860 0x555d7a3d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cd860 msgr2=0x555d7a3d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cd860 0x555d7a3d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 msgr2=0x555d7a3d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/613240032 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d7a39c780 con 0x555d7a3d29d0 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cd860 0x555d7a3d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:08.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a3d3fc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbe50061bd0 tx=0x7fbe50059420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df708fe130542a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe427fc640 1 -- 172.21.15.156:0/613240032 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50059520 con 0x555d7a3d29d0 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe427fc640 1 -- 172.21.15.156:0/613240032 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5005d8d0 con 0x555d7a3d29d0 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.669+0000 7fbe427fc640 1 -- 172.21.15.156:0/613240032 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50062100 con 0x555d7a3d29d0 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 -- 172.21.15.156:0/613240032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 msgr2=0x555d7a3d3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a3d3fc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbe50061bd0 tx=0x7fbe50059420 comp rx=0 tx=0).stop 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 -- 172.21.15.156:0/613240032 shutdown_connections 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a3d3fc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cdc40 0x555d7a3d3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 --2- 172.21.15.156:0/613240032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cd860 0x555d7a3d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 -- 172.21.15.156:0/613240032 >> 172.21.15.156:0/613240032 conn(0x555d7a3a0d40 msgr2=0x555d7a3e1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 -- 172.21.15.156:0/613240032 shutdown_connections 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 -- 172.21.15.156:0/613240032 wait complete. 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.670+0000 7fbe6201a740 1 -- start start 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 Processor -- start 2024-03-29T04:09:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cd860 0x555d7a3d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cdc40 0x555d7a3d1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a40e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555d7a3a5110 con 0x555d7a3d29d0 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d7a3e3130 con 0x555d7a3cd860 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x555d7a2c59c0 con 0x555d7a3cdc40 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.671+0000 7fbe6201a740 10 client.0 _ll_register_callbacks cb 0x7fbe440036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a40e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a40e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48838/0 (socket says 172.21.15.156:48838) 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/2706793464 learned_addr learned my addr 172.21.15.156:0/2706793464 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5c8ed640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cdc40 0x555d7a3d1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cd860 0x555d7a3d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cdc40 msgr2=0x555d7a3d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cdc40 0x555d7a3d1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cd860 msgr2=0x555d7a3d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:08.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cd860 0x555d7a3d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 -- 172.21.15.156:0/2706793464 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d7a3cd1d0 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe5d8ef640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a40e3e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbe5000ffd0 tx=0x7fbe5001f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974d3d17c29957ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe37fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe37fff640 1 -- 172.21.15.156:0/2706793464 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50059520 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe37fff640 1 -- 172.21.15.156:0/2706793464 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe50058970 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe5005d8d0 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe37fff640 1 -- 172.21.15.156:0/2706793464 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fbe50058b70 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe6201a740 10 client.4701 fetch_fsmap learned FSMap version 7 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.672+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x555d7a3ed440 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe37fff640 1 -- 172.21.15.156:0/2706793464 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5005d8d0 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe37fff640 1 -- 172.21.15.156:0/2706793464 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7fbe50071b10 con 0x555d7a3d29d0 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 10 client.4701 fetch_fsmap finished waiting for FSMap version 7 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 10 client.4701 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 20 client.4701 populate_metadata read hostname 'smithi156' 2024-03-29T04:09:08.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a40ea40 0x555d7a40ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 4 client.4701 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 --> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] -- command(tid 0: {"prefix": "status"}) -- 0x555d7a3118b0 con 0x555d7a40ea40 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a40ea40 0x555d7a40ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe37fff640 10 client.4701 ms_handle_connect on v2:172.21.15.156:6836/2168836037 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe5d0ee640 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a40ea40 0x555d7a40ee20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4853e79257c1f6fd server_cookie=fb8be1d2cb021427 in_seq=0 out_seq=0 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe37fff640 1 -- 172.21.15.156:0/2706793464 <== mds.0 v2:172.21.15.156:6836/2168836037 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x555d7a3118b0 con 0x555d7a40ea40 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe37fff640 10 client.4701 handle_command_reply: tid=0 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 client.4701 shutdown 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 msgr2=0x555d7a40e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a40e3e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbe5000ffd0 tx=0x7fbe5001f040 comp rx=0 tx=0).stop 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 shutdown_connections 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7a3d29d0 0x555d7a40e3e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x555d7a3cdc40 0x555d7a3d1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.156:6836/2168836037,v1:172.21.15.156:6837/2168836037] conn(0x555d7a40ea40 0x555d7a40ee20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 --2- 172.21.15.156:0/2706793464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d7a3cd860 0x555d7a3d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe37fff640 0 client.4701 ms_handle_reset on v2:172.21.15.156:6836/2168836037 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.673+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 >> 172.21.15.156:0/2706793464 conn(0x555d7a3a0d40 msgr2=0x555d7a3e1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 shutdown_connections 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe6201a740 1 -- 172.21.15.156:0/2706793464 wait complete. 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe6201a740 20 client.4701 trim_cache size 0 max 16384 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "6a348a99-5402-4006-8e25-31481aaa0bf5", 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 0, 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4290, 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-03-29T04:09:08.662 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-03-29T04:09:08.663 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 14.864407336999999, 2024-03-29T04:09:08.663 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 7, 2024-03-29T04:09:08.663 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 21, 2024-03-29T04:09:08.663 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 21, 2024-03-29T04:09:08.663 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 21.488722071000002 2024-03-29T04:09:08.663 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:09:08.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbe3c06c530 msgr2=0x7fbe3c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbe3c06c530 0x7fbe3c06e9f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbe54036000 tx=0x7fbe54030000 comp rx=0 tx=0).stop 2024-03-29T04:09:08.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 msgr2=0x7fbe5814d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 0x7fbe5814d090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbe500013d0 tx=0x7fbe5000e530 comp rx=0 tx=0).stop 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 shutdown_connections 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbe3c06c530 0x7fbe3c06e9f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe580ca8f0 0x7fbe5814d090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe580b3e70 0x7fbe580ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 --2- 172.21.15.156:0/865516067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbe440039b0 0x7fbe5814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 >> 172.21.15.156:0/865516067 conn(0x7fbe5809d7a0 msgr2=0x7fbe580d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 shutdown_connections 2024-03-29T04:09:08.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:08.674+0000 7fbe5f352640 1 -- 172.21.15.156:0/865516067 wait complete. 2024-03-29T04:09:08.666 INFO:teuthology.run_tasks:Running task sequential... 2024-03-29T04:09:08.679 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-29T04:09:08.679 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-29T04:09:08.679 INFO:teuthology.task.print:Enabling mgr modules 2024-03-29T04:09:08.680 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-29T04:09:08.731 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-29T04:09:08.731 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-29T04:09:08.731 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-29T04:09:08.731 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T04:09:08.731 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:08.731 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-29T04:09:08.731 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T04:09:08.731 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:08.731 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-29T04:09:08.731 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T04:09:08.732 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:08.732 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-29T04:09:08.732 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T04:09:08.732 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:08.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:08.732 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-03-29T04:09:08.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:08.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link delete ceph-brx 2024-03-29T04:09:08.824 INFO:teuthology.orchestra.run.smithi203.stderr:Cannot find device "ceph-brx" 2024-03-29T04:09:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:09:08.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:08.827 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-29T04:09:08.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:08.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2024-03-29T04:09:08.912 INFO:teuthology.orchestra.run.smithi156.stderr:Cannot find device "ceph-brx" 2024-03-29T04:09:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:09:08.915 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-29T04:09:08.916 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T04:09:09.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.077+0000 7fa92cfcd640 1 Processor -- start 2024-03-29T04:09:09.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.077+0000 7fa92cfcd640 1 -- start start 2024-03-29T04:09:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa92cfcd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 0x7fa9280cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa92cfcd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9280d5230 con 0x7fa92815ea20 2024-03-29T04:09:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa92cfcd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9280cf400 con 0x7fa9280d3120 2024-03-29T04:09:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa92cfcd640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280cfa50 con 0x7fa92815e640 2024-03-29T04:09:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa91ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa92815ea20 0x7fa9280d2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47656/0 (socket says 172.21.15.156:47656) 2024-03-29T04:09:09.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa91ffff640 1 -- 172.21.15.156:0/2218841427 learned_addr learned my addr 172.21.15.156:0/2218841427 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204319077 0 0) 0x7fa9280d5230 con 0x7fa92815ea20 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa92659c640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 0x7fa9280cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa914003520 con 0x7fa92815ea20 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1906040566 0 0) 0x7fa914003520 con 0x7fa92815ea20 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 msgr2=0x7fa9280cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 0x7fa9280cfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 >> v1:172.21.15.203:6789/0 conn(0x7fa9280d3120 legacy=0x7fa9280b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280cf6e0 con 0x7fa92815ea20 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fa910001670 con 0x7fa92815ea20 2024-03-29T04:09:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 >> v1:172.21.15.156:6789/0 conn(0x7fa92815ea20 legacy=0x7fa9280d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 0x7fa9140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa914004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914004d40 0x7fa9140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280cfa50 con 0x7fa914004010 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa91400bb80 con 0x7fa914004d40 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.078+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa91400bd80 con 0x7fa92815e640 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa914004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa92659c640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 0x7fa9140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa926d9d640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914004d40 0x7fa9140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 -- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 msgr2=0x7fa9140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa92815e640 0x7fa9140039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 -- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914004d40 msgr2=0x7fa9140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914004d40 0x7fa9140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 -- 172.21.15.156:0/2218841427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa914003520 con 0x7fa914004010 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.079+0000 7fa91ffff640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa914004800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c240 tx=0x7fa914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5af44fb152b1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910007d10 con 0x7fa914004010 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa910030070 con 0x7fa914004010 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa925d9b640 1 -- 172.21.15.156:0/2218841427 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91002cdc0 con 0x7fa914004010 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa92cfcd640 1 -- 172.21.15.156:0/2218841427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 msgr2=0x7fa914004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/2218841427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa914004800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c240 tx=0x7fa914005830 comp rx=0 tx=0).stop 2024-03-29T04:09:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa926d9d640 1 -- 172.21.15.156:0/2218841427 reap_dead start 2024-03-29T04:09:09.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa92cfcd640 1 -- 172.21.15.156:0/2218841427 shutdown_connections 2024-03-29T04:09:09.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa92cfcd640 1 -- 172.21.15.156:0/2218841427 >> 172.21.15.156:0/2218841427 conn(0x7fa92808bfd0 msgr2=0x7fa9280d4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:09.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa92cfcd640 1 -- 172.21.15.156:0/2218841427 shutdown_connections 2024-03-29T04:09:09.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.080+0000 7fa92cfcd640 1 -- 172.21.15.156:0/2218841427 wait complete. 2024-03-29T04:09:09.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.082+0000 7fa92cfcd640 1 Processor -- start 2024-03-29T04:09:09.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.082+0000 7fa92cfcd640 1 -- start start 2024-03-29T04:09:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.083+0000 7fa92cfcd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa9280cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.083+0000 7fa92cfcd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa928153020 0x7fa928159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.083+0000 7fa92cfcd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa92815a200 0x7fa9280b9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.083+0000 7fa92cfcd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280d24b0 con 0x7fa914004010 2024-03-29T04:09:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.083+0000 7fa92cfcd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280d2330 con 0x7fa92815a200 2024-03-29T04:09:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.083+0000 7fa92cfcd640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280d4d50 con 0x7fa928153020 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa9280cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa9280cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48856/0 (socket says 172.21.15.156:48856) 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 -- 172.21.15.156:0/1700740650 learned_addr learned my addr 172.21.15.156:0/1700740650 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa91ffff640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa928153020 0x7fa928159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa926d9d640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa92815a200 0x7fa9280b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 -- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa928153020 msgr2=0x7fa928159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa928153020 0x7fa928159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 -- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa92815a200 msgr2=0x7fa9280b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa92815a200 0x7fa9280b9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 -- 172.21.15.156:0/1700740650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280cc0b0 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92659c640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa9280cbb70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa91800b7f0 tx=0x7fa91800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918018070 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9180107d0 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9280b70c0 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918010ba0 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.084+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9280b7680 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.085+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa928068610 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.086+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa91801b070 con 0x7fa914004010 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.086+0000 7fa91effd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa90806c530 0x7fa90806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.086+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa91808bb90 con 0x7fa914004010 2024-03-29T04:09:09.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.086+0000 7fa91ffff640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa90806c530 0x7fa90806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:09.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.087+0000 7fa91ffff640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa90806c530 0x7fa90806e9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa910034000 tx=0x7fa910009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:09.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.091+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa918059bf0 con 0x7fa914004010 2024-03-29T04:09:09.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.309+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa9281616c0 con 0x7fa914004010 2024-03-29T04:09:09.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.309+0000 7fa91effd640 1 -- 172.21.15.156:0/1700740650 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7fa91805def0 con 0x7fa914004010 2024-03-29T04:09:09.289 INFO:teuthology.orchestra.run.smithi156.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa90806c530 msgr2=0x7fa90806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa90806c530 0x7fa90806e9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa910034000 tx=0x7fa910009030 comp rx=0 tx=0).stop 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 msgr2=0x7fa9280cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa9280cbb70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa91800b7f0 tx=0x7fa91800bcb0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 shutdown_connections 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa90806c530 0x7fa90806e9f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa92815a200 0x7fa9280b9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa928153020 0x7fa928159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 --2- 172.21.15.156:0/1700740650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa914004010 0x7fa9280cbb70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:09.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 >> 172.21.15.156:0/1700740650 conn(0x7fa92808bfd0 msgr2=0x7fa92815ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:09.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 shutdown_connections 2024-03-29T04:09:09.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09.311+0000 7fa92cfcd640 1 -- 172.21.15.156:0/1700740650 wait complete. 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T04:09:09.302 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T04:09:09.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:09.303 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:09.318 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42741sec preferred_lft 42741sec 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-03-29T04:09:09.319 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:09.319 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add name ceph-brx type bridge 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr flush dev ceph-brx 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set ceph-brx up 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-29T04:09:09.320 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:09.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:09.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:09.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:09.531 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T04:09:09.558 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-29T04:09:09.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:09.560 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-03-29T04:09:09.629 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-03-29T04:09:09.630 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T04:09:09.630 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T04:09:09.630 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T04:09:09.630 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T04:09:09.630 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:09.630 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:09.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T04:09:09.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T04:09:09.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T04:09:09.631 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:09.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:09.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:09.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:09.938 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-29T04:09:09.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:09.952 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2024-03-29T04:09:10.007 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:10.007 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:10.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:10.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-29T04:09:10.007 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:10.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:10.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:10.128 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-29T04:09:10.129 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:10.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:10.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:10.272 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:10.272 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:10.272 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2024-03-29T04:09:10.272 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.0 master ceph-brx 2024-03-29T04:09:10.272 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:10.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:10.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:10.359 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-29T04:09:10.359 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:10.359 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:10.373 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T04:09:10.373 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:10.374 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:10.428 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-29T04:09:10.496 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:10.552 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-29T04:09:10.552 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-29T04:09:10.552 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-29T04:09:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-0 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-1 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-2 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-3 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:10.554 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:10.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:10.555 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-29T04:09:10.608 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-29T04:09:10.608 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-29T04:09:10.651 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-29T04:09:10.678 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-29T04:09:10.736 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-29T04:09:10.737 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-0 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-1 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-2 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-3 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:10.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:10.738 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-29T04:09:10.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:2024-03-29T04:09:10.759+0000 7fb7790da480 -1 init, newargv = 0x556327f98a50 newargc=15 2024-03-29T04:09:10.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:ceph-fuse[46124]: starting ceph client 2024-03-29T04:09:10.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:ceph-fuse[46124]: starting fuse 2024-03-29T04:09:10.754 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-29T04:09:10.755 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-29T04:09:10.755 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-29T04:09:10.756 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:10.757 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.0")) 2024-03-29T04:09:10.757 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-29T04:09:10.876 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.0.46124.asok 2024-03-29T04:09:10.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:10.887 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46124.asok 2024-03-29T04:09:10.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:10.887 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46124.asok status 2024-03-29T04:09:11.024 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:09:11.024 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-03-29T04:09:11.024 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "69e2cd73c52284dfd5096c66b60c6e6b26a3c447", 2024-03-29T04:09:11.024 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2520-g69e2cd73 (69e2cd73c52284dfd5096c66b60c6e6b26a3c447) squid (dev)", 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "0", 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "46124", 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4716, 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-29T04:09:11.025 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4716 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 1822575599 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 1822575599 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:11.026 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4716 192.168.0.1:0/1822575599", 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/1822575599", 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 7, 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 21, 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-03-29T04:09:11.027 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:09:11.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T04:09:11.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.156+0000 7efd5e785640 1 Processor -- start 2024-03-29T04:09:11.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.156+0000 7efd5e785640 1 -- start start 2024-03-29T04:09:11.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd5e785640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7efd5815e730 0x7efd580ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd5e785640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd580cfa20 con 0x7efd580d49d0 2024-03-29T04:09:11.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd5e785640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd580cfd00 con 0x7efd580cf3e0 2024-03-29T04:09:11.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd5e785640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7efd5806c720 con 0x7efd5815e730 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd5cd22640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7efd580d49d0 0x7efd5806cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47658/0 (socket says 172.21.15.156:47658) 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd5cd22640 1 -- 172.21.15.156:0/2802234023 learned_addr learned my addr 172.21.15.156:0/2802234023 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd57fff640 1 --2- 172.21.15.156:0/2802234023 >> v2:172.21.15.203:3300/0 conn(0x7efd5815e730 0x7efd580ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.157+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458304741 0 0) 0x7efd580cfa20 con 0x7efd580d49d0 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd40003520 con 0x7efd580d49d0 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2572152387 0 0) 0x7efd40003520 con 0x7efd580d49d0 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 >> v2:172.21.15.203:3300/0 conn(0x7efd5815e730 msgr2=0x7efd580ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 --2- 172.21.15.156:0/2802234023 >> v2:172.21.15.203:3300/0 conn(0x7efd5815e730 0x7efd580ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 >> v1:172.21.15.203:6789/0 conn(0x7efd580cf3e0 legacy=0x7efd580d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd580cffe0 con 0x7efd580d49d0 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7efd500019f0 con 0x7efd580d49d0 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 >> v1:172.21.15.156:6789/0 conn(0x7efd580d49d0 legacy=0x7efd5806cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400039b0 0x7efd40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd40004800 0x7efd40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd400051c0 0x7efd4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efd5806c720 con 0x7efd400051c0 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efd4000c160 con 0x7efd40004800 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7efd4000c360 con 0x7efd400039b0 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd5cd22640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd400051c0 0x7efd4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd577fe640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd40004800 0x7efd40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd57fff640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400039b0 0x7efd40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd5cd22640 1 -- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400039b0 msgr2=0x7efd40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd5cd22640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400039b0 0x7efd40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd5cd22640 1 -- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd40004800 msgr2=0x7efd40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd5cd22640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd40004800 0x7efd40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.158+0000 7efd5cd22640 1 -- 172.21.15.156:0/2802234023 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd40003520 con 0x7efd400051c0 2024-03-29T04:09:11.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.159+0000 7efd5cd22640 1 -- 172.21.15.156:0/2802234023 reap_dead start 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.159+0000 7efd5cd22640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd400051c0 0x7efd4000b9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd5000a840 tx=0x7efd40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5fcb457eb58f7e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.159+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5000a460 con 0x7efd400051c0 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.159+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd50034650 con 0x7efd400051c0 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.159+0000 7efd56ffd640 1 -- 172.21.15.156:0/2802234023 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50034a40 con 0x7efd400051c0 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 -- 172.21.15.156:0/2802234023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd400051c0 msgr2=0x7efd4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd400051c0 0x7efd4000b9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd5000a840 tx=0x7efd40005830 comp rx=0 tx=0).stop 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 -- 172.21.15.156:0/2802234023 shutdown_connections 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 --2- 172.21.15.156:0/2802234023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd400051c0 0x7efd4000b9c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 -- 172.21.15.156:0/2802234023 >> 172.21.15.156:0/2802234023 conn(0x7efd5808bfd0 msgr2=0x7efd5809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 -- 172.21.15.156:0/2802234023 shutdown_connections 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.160+0000 7efd5e785640 1 -- 172.21.15.156:0/2802234023 wait complete. 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.161+0000 7efd5e785640 1 Processor -- start 2024-03-29T04:09:11.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.161+0000 7efd5e785640 1 -- start start 2024-03-29T04:09:11.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5e785640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400051c0 0x7efd580cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5e785640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 0x7efd580cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5e785640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd580cd5a0 0x7efd580cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5e785640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efd5815eb10 con 0x7efd580b75b0 2024-03-29T04:09:11.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5e785640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efd580d2220 con 0x7efd580cd5a0 2024-03-29T04:09:11.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5e785640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7efd5815ec90 con 0x7efd400051c0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 0x7efd580cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 0x7efd580cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48872/0 (socket says 172.21.15.156:48872) 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 -- 172.21.15.156:0/3849102160 learned_addr learned my addr 172.21.15.156:0/3849102160 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd5cd22640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd580cd5a0 0x7efd580cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd57fff640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400051c0 0x7efd580cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 -- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400051c0 msgr2=0x7efd580cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400051c0 0x7efd580cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 -- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd580cd5a0 msgr2=0x7efd580cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd580cd5a0 0x7efd580cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 -- 172.21.15.156:0/3849102160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd580b7b70 con 0x7efd580b75b0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd57fff640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400051c0 0x7efd580cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.162+0000 7efd577fe640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 0x7efd580cd060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd4c001f00 tx=0x7efd4c00fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.163+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c01a070 con 0x7efd580b75b0 2024-03-29T04:09:11.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.163+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd4c005320 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.163+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c006a40 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.163+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd580b7e50 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.163+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd580b8410 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.164+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd58068610 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.164+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7efd4c01d070 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.164+0000 7efd54ff9640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7efd4406c560 0x7efd4406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.165+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efd4c08adb0 con 0x7efd580b75b0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.165+0000 7efd57fff640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7efd4406c560 0x7efd4406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.165+0000 7efd57fff640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7efd4406c560 0x7efd4406ea20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd48009b40 tx=0x7efd48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:11.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.169+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7efd4c058e10 con 0x7efd580b75b0 2024-03-29T04:09:11.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.459+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efd5806d4b0 con 0x7efd580b75b0 2024-03-29T04:09:11.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.459+0000 7efd54ff9640 1 -- 172.21.15.156:0/3849102160 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7efd4c05d110 con 0x7efd580b75b0 2024-03-29T04:09:11.440 INFO:teuthology.orchestra.run.smithi156.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T04:09:11.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7efd4406c560 msgr2=0x7efd4406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7efd4406c560 0x7efd4406ea20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd48009b40 tx=0x7efd48008040 comp rx=0 tx=0).stop 2024-03-29T04:09:11.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 msgr2=0x7efd580cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:11.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 0x7efd580cd060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd4c001f00 tx=0x7efd4c00fa50 comp rx=0 tx=0).stop 2024-03-29T04:09:11.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 shutdown_connections 2024-03-29T04:09:11.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7efd4406c560 0x7efd4406ea20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd580cd5a0 0x7efd580cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd580b75b0 0x7efd580cd060 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 --2- 172.21.15.156:0/3849102160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd400051c0 0x7efd580cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 >> 172.21.15.156:0/3849102160 conn(0x7efd5808bfd0 msgr2=0x7efd580d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 shutdown_connections 2024-03-29T04:09:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11.461+0000 7efd5e785640 1 -- 172.21.15.156:0/3849102160 wait complete. 2024-03-29T04:09:11.452 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T04:09:11.452 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T04:09:11.452 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-03-29T04:09:11.452 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-29T04:09:11.453 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:11.453 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T04:09:11.453 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T04:09:11.453 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T04:09:11.453 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-29T04:09:11.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.454 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-03-29T04:09:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:11.475 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:11.475 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:11.475 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.475 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:11.475 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.476 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:11.477 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:11.477 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-03-29T04:09:11.477 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:11.477 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:11.478 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-03-29T04:09:11.478 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:11.478 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:11.479 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42739sec preferred_lft 42739sec 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:09:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 32:dc:33:e3:68:e4 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::6075:d6ff:fee6:8b38/64 scope link 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 32:dc:33:e3:68:e4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:11.481 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::30dc:33ff:fee3:68e4/64 scope link tentative 2024-03-29T04:09:11.482 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.482 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T04:09:11.541 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-29T04:09:11.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.543 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-03-29T04:09:11.599 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-03-29T04:09:11.600 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T04:09:11.600 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T04:09:11.600 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T04:09:11.600 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T04:09:11.600 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:11.600 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:11.600 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T04:09:11.600 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T04:09:11.600 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T04:09:11.601 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:11.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:11.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:11.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.738 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-29T04:09:11.753 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T04:09:11.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.753 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2024-03-29T04:09:11.808 INFO:teuthology.orchestra.run.smithi156.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-29T04:09:11.808 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:11.808 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:11.808 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:09:11.808 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-29T04:09:11.808 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:11.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:11.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:11.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.931 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-29T04:09:11.957 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:11.958 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:11.958 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:11.958 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.958 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:11.958 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.958 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:09:11.959 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0a:38:18:ad:66:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T04:09:11.959 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T04:09:11.959 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.959 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::838:18ff:fead:6603/64 scope link tentative 2024-03-29T04:09:11.959 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:11.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:11.961 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-29T04:09:12.028 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:12.028 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0a:38:18:ad:66:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::838:18ff:fead:6603/64 scope link 2024-03-29T04:09:12.029 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:12.030 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-29T04:09:12.031 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:12.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:12.192 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:12.200 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:12.200 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:09:12.200 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.1 up 2024-03-29T04:09:12.200 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.1 master ceph-brx 2024-03-29T04:09:12.200 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:09:12.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:12.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:12.284 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-29T04:09:12.285 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:12.285 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:12.299 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-29T04:09:12.299 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:12.299 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:12.353 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-29T04:09:12.420 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-29T04:09:12.476 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-29T04:09:12.477 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-0 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-1 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-2 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-3 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:09:12.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:12.478 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-29T04:09:12.530 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-29T04:09:12.530 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-29T04:09:12.531 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-29T04:09:12.572 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-29T04:09:12.599 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:12.655 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-29T04:09:12.655 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-29T04:09:12.656 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-0 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-1 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-2 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-3 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-29T04:09:12.657 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:12.658 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:09:12.658 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:12.658 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:09:12.658 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:09:12.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:12.658 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-29T04:09:12.659 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi156.stderr:2024-03-29T04:09:12.680+0000 7feae3a36480 -1 init, newargv = 0x55a5ec9ea300 newargc=15 2024-03-29T04:09:12.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi156.stderr:ceph-fuse[46622]: starting ceph client 2024-03-29T04:09:12.664 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi156.stderr:ceph-fuse[46622]: starting fuse 2024-03-29T04:09:12.675 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-29T04:09:12.675 INFO:teuthology.orchestra.run.smithi156.stdout:44 2024-03-29T04:09:12.676 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-29T04:09:12.676 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.2")) 2024-03-29T04:09:12.677 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-29T04:09:12.796 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.2.46622.asok 2024-03-29T04:09:12.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:12.806 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.46622.asok 2024-03-29T04:09:12.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:12.806 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.46622.asok status 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "69e2cd73c52284dfd5096c66b60c6e6b26a3c447", 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2520-g69e2cd73 (69e2cd73c52284dfd5096c66b60c6e6b26a3c447) squid (dev)", 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "2", 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "46622", 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-03-29T04:09:12.944 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4728, 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4728 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2785117090 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:12.945 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2785117090 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4728 192.168.0.2:0/2785117090", 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.2:0/2785117090", 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 7, 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 21, 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-03-29T04:09:12.946 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:09:12.952 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T04:09:13.055 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.075+0000 7f66600d9640 1 Processor -- start 2024-03-29T04:09:13.055 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.075+0000 7f66600d9640 1 -- start start 2024-03-29T04:09:13.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f66600d9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66580720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:13.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f66600d9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f66580725f0 0x7f665815e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:13.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f66600d9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6658072a50 con 0x7f665815ab40 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f66600d9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f66580b4d90 con 0x7f66580725f0 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f66600d9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f665806e020 con 0x7f66580ad480 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f665de75640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f665815ab40 0x7f66580b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47096/0 (socket says 172.21.15.156:47096) 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f665de75640 1 -- 172.21.15.156:0/1913611241 learned_addr learned my addr 172.21.15.156:0/1913611241 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f665d674640 1 --2- 172.21.15.156:0/1913611241 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66580720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.076+0000 7f665e676640 1 --2- 172.21.15.156:0/1913611241 >> v2:172.21.15.203:3300/0 conn(0x7f66580725f0 0x7f665815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665ce73640 1 -- 172.21.15.156:0/1913611241 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464992082 0 0) 0x7f6658072a50 con 0x7f665815ab40 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665ce73640 1 -- 172.21.15.156:0/1913611241 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6644003520 con 0x7f665815ab40 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665d674640 1 -- 172.21.15.156:0/1913611241 >> v2:172.21.15.203:3300/0 conn(0x7f66580725f0 msgr2=0x7f665815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665d674640 1 --2- 172.21.15.156:0/1913611241 >> v2:172.21.15.203:3300/0 conn(0x7f66580725f0 0x7f665815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665d674640 1 -- 172.21.15.156:0/1913611241 >> v1:172.21.15.156:6789/0 conn(0x7f665815ab40 legacy=0x7f66580b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665ce73640 1 -- 172.21.15.156:0/1913611241 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 970363988 0 0) 0x7f6644003520 con 0x7f665815ab40 2024-03-29T04:09:13.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.077+0000 7f665d674640 1 -- 172.21.15.156:0/1913611241 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665806e300 con 0x7f66580ad480 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f665d674640 1 --2- 172.21.15.156:0/1913611241 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66580720b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f664800ba40 tx=0x7f664802ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22a0aa4db99789b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f665ce73640 1 -- 172.21.15.156:0/1913611241 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664803b070 con 0x7f66580ad480 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f665ce73640 1 -- 172.21.15.156:0/1913611241 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664802ea00 con 0x7f66580ad480 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f665ce73640 1 -- 172.21.15.156:0/1913611241 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650001670 con 0x7f66580ad480 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 -- 172.21.15.156:0/1913611241 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 msgr2=0x7f66580720b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 --2- 172.21.15.156:0/1913611241 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66580720b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f664800ba40 tx=0x7f664802ded0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 -- 172.21.15.156:0/1913611241 shutdown_connections 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 --2- 172.21.15.156:0/1913611241 >> v2:172.21.15.203:3300/0 conn(0x7f66580725f0 0x7f665815e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 --2- 172.21.15.156:0/1913611241 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66580720b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 -- 172.21.15.156:0/1913611241 >> 172.21.15.156:0/1913611241 conn(0x7f66580b1d50 msgr2=0x7f66580b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:13.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 -- 172.21.15.156:0/1913611241 shutdown_connections 2024-03-29T04:09:13.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.078+0000 7f66600d9640 1 -- 172.21.15.156:0/1913611241 wait complete. 2024-03-29T04:09:13.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 Processor -- start 2024-03-29T04:09:13.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 -- start start 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 0x7f66581562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66581506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f665815ab40 0x7f6658150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66580b4a50 con 0x7f66580725f0 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f66580b5070 con 0x7f665815ab40 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f66600d9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6658071d00 con 0x7f66580ad480 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f665de75640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 0x7f66581562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f665de75640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 0x7f66581562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43676/0 (socket says 172.21.15.156:43676) 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f665de75640 1 -- 172.21.15.156:0/3770528516 learned_addr learned my addr 172.21.15.156:0/3770528516 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f665d674640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66581506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.080+0000 7f665e676640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f665815ab40 0x7f6658150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:13.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f665de75640 1 -- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 msgr2=0x7f66581506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f665de75640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66581506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f665de75640 1 -- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f665815ab40 msgr2=0x7f6658150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f665de75640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f665815ab40 0x7f6658150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f665de75640 1 -- 172.21.15.156:0/3770528516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6658156dc0 con 0x7f66580725f0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f665de75640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 0x7f66581562e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f66500017e0 tx=0x7f665000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650005ad0 con 0x7f66580725f0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6650005c90 con 0x7f66580725f0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66581570a0 con 0x7f66580725f0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66500072b0 con 0x7f66580725f0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.081+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6658157660 con 0x7f66580725f0 2024-03-29T04:09:13.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.082+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6658157990 con 0x7f66580725f0 2024-03-29T04:09:13.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.083+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6650007450 con 0x7f66580725f0 2024-03-29T04:09:13.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.083+0000 7f664e7fc640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f663c06c530 0x7f663c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:13.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.083+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f665008b350 con 0x7f66580725f0 2024-03-29T04:09:13.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.083+0000 7f665d674640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f663c06c530 0x7f663c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:13.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.084+0000 7f665d674640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f663c06c530 0x7f663c06e9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6648038000 tx=0x7f6648032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:13.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.087+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f66500593b0 con 0x7f66580725f0 2024-03-29T04:09:13.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.308+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6658068610 con 0x7f66580725f0 2024-03-29T04:09:13.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.308+0000 7f664e7fc640 1 -- 172.21.15.156:0/3770528516 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f665005d6b0 con 0x7f66580725f0 2024-03-29T04:09:13.288 INFO:teuthology.orchestra.run.smithi156.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f663c06c530 msgr2=0x7f663c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f663c06c530 0x7f663c06e9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6648038000 tx=0x7f6648032000 comp rx=0 tx=0).stop 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 msgr2=0x7f66581562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 0x7f66581562e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f66500017e0 tx=0x7f665000cbe0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 shutdown_connections 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f663c06c530 0x7f663c06e9f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f665815ab40 0x7f6658150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f66580ad480 0x7f66581506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 --2- 172.21.15.156:0/3770528516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66580725f0 0x7f66581562e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 >> 172.21.15.156:0/3770528516 conn(0x7f66580b1d50 msgr2=0x7f665814d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 shutdown_connections 2024-03-29T04:09:13.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:13.310+0000 7f66600d9640 1 -- 172.21.15.156:0/3770528516 wait complete. 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi203.front.sepia.ceph.com 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T04:09:13.300 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-29T04:09:13.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:13.301 DEBUG:teuthology.orchestra.run.smithi203:> ip addr 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:44 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:13.320 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f0 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:45 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f1 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:22 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:23 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: inet 172.21.15.203/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft 42734sec preferred_lft 42734sec 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::ec4:7aff:fe8f:4a23/64 scope link 2024-03-29T04:09:13.321 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:13.321 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add name ceph-brx type bridge 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip addr flush dev ceph-brx 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set ceph-brx up 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-29T04:09:13.322 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:13.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:13.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:13.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:13.680 DEBUG:teuthology.orchestra.run.smithi203:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T04:09:13.709 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-29T04:09:13.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:13.711 DEBUG:teuthology.orchestra.run.smithi203:> route 2024-03-29T04:09:13.828 INFO:teuthology.orchestra.run.smithi203.stdout:Kernel IP routing table 2024-03-29T04:09:13.828 INFO:teuthology.orchestra.run.smithi203.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T04:09:13.828 INFO:teuthology.orchestra.run.smithi203.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T04:09:13.829 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T04:09:13.829 INFO:teuthology.orchestra.run.smithi203.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T04:09:13.829 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:13.829 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:13.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T04:09:13.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T04:09:13.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T04:09:13.829 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:13.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:14.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:14.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:14.100 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-03-29T04:09:14.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:14.115 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list-id 2024-03-29T04:09:14.170 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:14.170 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:14.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:14.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-29T04:09:14.170 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:14.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:14.291 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-29T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:14.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:14.422 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:14.428 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:14.428 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:14.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set brx.0 up 2024-03-29T04:09:14.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set dev brx.0 master ceph-brx 2024-03-29T04:09:14.428 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:14.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:14.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:14.510 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-29T04:09:14.510 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:14.510 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:14.524 INFO:teuthology.orchestra.run.smithi203.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-29T04:09:14.525 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:14.525 DEBUG:teuthology.orchestra.run.smithi203:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:14.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-03-29T04:09:14.647 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:14.703 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:14.704 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2024-03-29T04:09:14.705 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2024-03-29T04:09:14.706 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2024-03-29T04:09:14.706 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2024-03-29T04:09:14.706 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-03-29T04:09:14.706 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:14.706 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:14.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:14.706 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-03-29T04:09:14.759 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-29T04:09:14.759 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-29T04:09:14.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-03-29T04:09:14.828 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:14.883 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-03-29T04:09:14.883 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-03-29T04:09:14.883 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-03-29T04:09:14.883 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:14.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:14.886 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-03-29T04:09:14.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:2024-03-29T04:09:14.907+0000 7f443403c480 -1 init, newargv = 0x55fede6a28f0 newargc=15 2024-03-29T04:09:14.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[43541]: starting ceph client 2024-03-29T04:09:14.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[43541]: starting fuse 2024-03-29T04:09:14.902 INFO:teuthology.orchestra.run.smithi203.stdout:41 2024-03-29T04:09:14.902 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-29T04:09:14.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo stdin-killer -- python3 -c ' 2024-03-29T04:09:14.902 DEBUG:teuthology.orchestra.run.smithi203:> import glob 2024-03-29T04:09:14.902 DEBUG:teuthology.orchestra.run.smithi203:> import re 2024-03-29T04:09:14.902 DEBUG:teuthology.orchestra.run.smithi203:> import os 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> import subprocess 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> def _find_admin_socket(client_name): 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> files = glob.glob(asok_path) 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> # Given a non-glob path, it better be there 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> if "*" not in asok_path: 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> assert(len(files) == 1) 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> return files[0] 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> for f in files: 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-29T04:09:14.903 DEBUG:teuthology.orchestra.run.smithi203:> contents = proc_f.read() 2024-03-29T04:09:14.904 DEBUG:teuthology.orchestra.run.smithi203:> if mountpoint in contents: 2024-03-29T04:09:14.904 DEBUG:teuthology.orchestra.run.smithi203:> return f 2024-03-29T04:09:14.904 DEBUG:teuthology.orchestra.run.smithi203:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-29T04:09:14.904 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:14.904 DEBUG:teuthology.orchestra.run.smithi203:> print(_find_admin_socket("client.1")) 2024-03-29T04:09:14.904 DEBUG:teuthology.orchestra.run.smithi203:> ' 2024-03-29T04:09:15.022 INFO:teuthology.orchestra.run.smithi203.stdout:/var/run/ceph/ceph-client.1.43541.asok 2024-03-29T04:09:15.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:15.032 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43541.asok 2024-03-29T04:09:15.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43541.asok status 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout: "metadata": { 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_sha1": "69e2cd73c52284dfd5096c66b60c6e6b26a3c447", 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2520-g69e2cd73 (69e2cd73c52284dfd5096c66b60c6e6b26a3c447) squid (dev)", 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout: "entity_id": "1", 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout: "hostname": "smithi203", 2024-03-29T04:09:15.168 INFO:teuthology.orchestra.run.smithi203.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "pid": "43541", 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "root": "/" 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_count": 0, 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_pinned_count": 0, 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "id": 4621, 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "inst": { 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "name": { 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "client", 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "num": 4621 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-03-29T04:09:15.169 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 3740322759 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 3740322759 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "inst_str": "client.4621 192.168.0.1:0/3740322759", 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "addr_str": "192.168.0.1:0/3740322759", 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "inode_count": 1, 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_epoch": 7, 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch": 21, 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch_barrier": 0, 2024-03-29T04:09:15.170 INFO:teuthology.orchestra.run.smithi203.stdout: "blocklisted": false, 2024-03-29T04:09:15.171 INFO:teuthology.orchestra.run.smithi203.stdout: "fs_name": "cephfs" 2024-03-29T04:09:15.171 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-29T04:09:15.176 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T04:09:15.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 Processor -- start 2024-03-29T04:09:15.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 -- start start 2024-03-29T04:09:15.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f202415c4f0 0x7f2024160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2024162230 0x7f202409b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f20240703c0 0x7f2024070850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2024070d90 con 0x7f20240703c0 2024-03-29T04:09:15.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f202406fc80 con 0x7f202415c4f0 2024-03-29T04:09:15.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.300+0000 7f2029300640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f20241627a0 con 0x7f2024162230 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f20240703c0 0x7f2024070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f20240703c0 0x7f2024070850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43680/0 (socket says 172.21.15.156:43680) 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 -- 172.21.15.156:0/4288046026 learned_addr learned my addr 172.21.15.156:0/4288046026 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f2022ffd640 1 --2- 172.21.15.156:0/4288046026 >> v2:172.21.15.203:3300/0 conn(0x7f202415c4f0 0x7f2024160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20227fc640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2024162230 0x7f202409b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 -- 172.21.15.156:0/4288046026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2024162230 msgr2=0x7f202409b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2024162230 0x7f202409b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 -- 172.21.15.156:0/4288046026 >> v2:172.21.15.203:3300/0 conn(0x7f202415c4f0 msgr2=0x7f2024160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 --2- 172.21.15.156:0/4288046026 >> v2:172.21.15.203:3300/0 conn(0x7f202415c4f0 0x7f2024160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20237fe640 1 -- 172.21.15.156:0/4288046026 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2024162a80 con 0x7f20240703c0 2024-03-29T04:09:15.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.301+0000 7f20227fc640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2024162230 0x7f202409b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f20237fe640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20240703c0 0x7f2024070850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2024073010 tx=0x7f201402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afc6158b2c67520 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2021ffb640 1 -- 172.21.15.156:0/4288046026 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014035020 con 0x7f20240703c0 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2021ffb640 1 -- 172.21.15.156:0/4288046026 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2014031870 con 0x7f20240703c0 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2021ffb640 1 -- 172.21.15.156:0/4288046026 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014031c40 con 0x7f20240703c0 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 -- 172.21.15.156:0/4288046026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20240703c0 msgr2=0x7f2024070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20240703c0 0x7f2024070850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2024073010 tx=0x7f201402b910 comp rx=0 tx=0).stop 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 -- 172.21.15.156:0/4288046026 shutdown_connections 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20240703c0 0x7f2024070850 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 --2- 172.21.15.156:0/4288046026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2024162230 0x7f202409b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 --2- 172.21.15.156:0/4288046026 >> v2:172.21.15.203:3300/0 conn(0x7f202415c4f0 0x7f2024160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 -- 172.21.15.156:0/4288046026 >> 172.21.15.156:0/4288046026 conn(0x7f20240d2250 msgr2=0x7f20240d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:15.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 -- 172.21.15.156:0/4288046026 shutdown_connections 2024-03-29T04:09:15.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.302+0000 7f2029300640 1 -- 172.21.15.156:0/4288046026 wait complete. 2024-03-29T04:09:15.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 Processor -- start 2024-03-29T04:09:15.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 -- start start 2024-03-29T04:09:15.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20240703c0 0x7f2024157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 0x7f20240c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024162230 0x7f20240c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20240d0420 con 0x7f202415c4f0 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2024163040 con 0x7f2024162230 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.304+0000 7f2029300640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f20240d05a0 con 0x7f20240703c0 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 0x7f20240c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 0x7f20240c5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43690/0 (socket says 172.21.15.156:43690) 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 -- 172.21.15.156:0/2089880994 learned_addr learned my addr 172.21.15.156:0/2089880994 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f2022ffd640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20240703c0 0x7f2024157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20237fe640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024162230 0x7f20240c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 -- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20240703c0 msgr2=0x7f2024157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:15.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20240703c0 0x7f2024157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 -- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024162230 msgr2=0x7f20240c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024162230 0x7f20240c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 -- 172.21.15.156:0/2089880994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2024157dc0 con 0x7f202415c4f0 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f2022ffd640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20240703c0 0x7f2024157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20237fe640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024162230 0x7f20240c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f20227fc640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 0x7f20240c5d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f201800c7f0 tx=0x7f201800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018015070 con 0x7f202415c4f0 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2018007a30 con 0x7f202415c4f0 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20240c68b0 con 0x7f202415c4f0 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018007e00 con 0x7f202415c4f0 2024-03-29T04:09:15.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.305+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20240b5e80 con 0x7f202415c4f0 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.306+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2024068610 con 0x7f202415c4f0 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.307+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2018018070 con 0x7f202415c4f0 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.307+0000 7f200b7fe640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f201006c530 0x7f201006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.307+0000 7f2022ffd640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f201006c530 0x7f201006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.307+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f201808b5f0 con 0x7f202415c4f0 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.308+0000 7f2022ffd640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f201006c530 0x7f201006e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f20241620d0 tx=0x7f201c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:15.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.312+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2018059c10 con 0x7f202415c4f0 2024-03-29T04:09:15.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.531+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f20240d3160 con 0x7f202415c4f0 2024-03-29T04:09:15.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.531+0000 7f200b7fe640 1 -- 172.21.15.156:0/2089880994 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f2018059510 con 0x7f202415c4f0 2024-03-29T04:09:15.511 INFO:teuthology.orchestra.run.smithi156.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f201006c530 msgr2=0x7f201006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f201006c530 0x7f201006e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f20241620d0 tx=0x7f201c00b040 comp rx=0 tx=0).stop 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 msgr2=0x7f20240c5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 0x7f20240c5d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f201800c7f0 tx=0x7f201800cbf0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 shutdown_connections 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f201006c530 0x7f201006e9f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024162230 0x7f20240c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f202415c4f0 0x7f20240c5d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 --2- 172.21.15.156:0/2089880994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20240703c0 0x7f2024157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:15.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 >> 172.21.15.156:0/2089880994 conn(0x7f20240d2250 msgr2=0x7f202406d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:15.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 shutdown_connections 2024-03-29T04:09:15.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:15.534+0000 7f2029300640 1 -- 172.21.15.156:0/2089880994 wait complete. 2024-03-29T04:09:15.524 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T04:09:15.524 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T04:09:15.524 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi203.front.sepia.ceph.com 2024-03-29T04:09:15.524 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-29T04:09:15.525 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:15.525 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T04:09:15.525 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T04:09:15.525 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T04:09:15.525 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-29T04:09:15.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.525 DEBUG:teuthology.orchestra.run.smithi203:> ip addr 2024-03-29T04:09:15.541 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:15.541 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:15.541 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:15.541 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:15.541 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:15.541 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:44 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f0 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:45 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f1 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:22 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:15.542 INFO:teuthology.orchestra.run.smithi203.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:23 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: inet 172.21.15.203/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft 42732sec preferred_lft 42732sec 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::ec4:7aff:fe8f:4a23/64 scope link 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 8a:cf:e6:65:5e:24 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::b881:18ff:fe2d:cce1/64 scope link 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T04:09:15.543 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 8a:cf:e6:65:5e:24 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:15.544 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::88cf:e6ff:fe65:5e24/64 scope link tentative 2024-03-29T04:09:15.544 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:15.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.544 DEBUG:teuthology.orchestra.run.smithi203:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T04:09:15.608 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-29T04:09:15.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.609 DEBUG:teuthology.orchestra.run.smithi203:> route 2024-03-29T04:09:15.666 INFO:teuthology.orchestra.run.smithi203.stdout:Kernel IP routing table 2024-03-29T04:09:15.666 INFO:teuthology.orchestra.run.smithi203.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T04:09:15.666 INFO:teuthology.orchestra.run.smithi203.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T04:09:15.666 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T04:09:15.666 INFO:teuthology.orchestra.run.smithi203.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T04:09:15.666 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:15.666 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:15.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T04:09:15.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T04:09:15.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T04:09:15.667 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:15.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:15.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:15.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.802 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-03-29T04:09:15.816 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-29T04:09:15.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.817 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list-id 2024-03-29T04:09:15.872 INFO:teuthology.orchestra.run.smithi203.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-29T04:09:15.873 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:15.873 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:15.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:09:15.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-29T04:09:15.873 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:15.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:15.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:15.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:15.995 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 9e:d0:e4:b5:15:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T04:09:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T04:09:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::9cd0:e4ff:feb5:156e/64 scope link 2024-03-29T04:09:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:16.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-29T04:09:16.090 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 9e:d0:e4:b5:15:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.091 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::9cd0:e4ff:feb5:156e/64 scope link 2024-03-29T04:09:16.092 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:09:16.093 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-29T04:09:16.093 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:16.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:16.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:16.257 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:09:16.257 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:09:16.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set brx.1 up 2024-03-29T04:09:16.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set dev brx.1 master ceph-brx 2024-03-29T04:09:16.258 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:09:16.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:09:16.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:16.339 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-29T04:09:16.339 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:16.339 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:16.354 INFO:teuthology.orchestra.run.smithi203.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-29T04:09:16.354 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:09:16.354 DEBUG:teuthology.orchestra.run.smithi203:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:16.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-03-29T04:09:16.476 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-03-29T04:09:16.532 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-03-29T04:09:16.533 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:09:16.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:16.535 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-03-29T04:09:16.588 INFO:teuthology.orchestra.run.smithi203.stdout:41 2024-03-29T04:09:16.588 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-29T04:09:16.588 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-29T04:09:16.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-03-29T04:09:16.657 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-03-29T04:09:16.714 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-03-29T04:09:16.715 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:09:16.716 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:09:16.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:16.717 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-03-29T04:09:16.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi203.stderr:2024-03-29T04:09:16.736+0000 7f9638eb5480 -1 init, newargv = 0x55db04ebd550 newargc=15 2024-03-29T04:09:16.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi203.stderr:ceph-fuse[43997]: starting ceph client 2024-03-29T04:09:16.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi203.stderr:ceph-fuse[43997]: starting fuse 2024-03-29T04:09:16.732 INFO:teuthology.orchestra.run.smithi203.stdout:41 2024-03-29T04:09:16.732 INFO:teuthology.orchestra.run.smithi203.stdout:44 2024-03-29T04:09:16.732 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo stdin-killer -- python3 -c ' 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> import glob 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> import re 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> import os 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> import subprocess 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> def _find_admin_socket(client_name): 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> files = glob.glob(asok_path) 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> # Given a non-glob path, it better be there 2024-03-29T04:09:16.733 DEBUG:teuthology.orchestra.run.smithi203:> if "*" not in asok_path: 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> assert(len(files) == 1) 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> return files[0] 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> for f in files: 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> contents = proc_f.read() 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> if mountpoint in contents: 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> return f 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> print(_find_admin_socket("client.3")) 2024-03-29T04:09:16.734 DEBUG:teuthology.orchestra.run.smithi203:> ' 2024-03-29T04:09:16.854 INFO:teuthology.orchestra.run.smithi203.stdout:/var/run/ceph/ceph-client.3.43997.asok 2024-03-29T04:09:16.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:09:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:09:16.865 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43997.asok 2024-03-29T04:09:16.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:16.865 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43997.asok status 2024-03-29T04:09:17.001 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "metadata": { 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_sha1": "69e2cd73c52284dfd5096c66b60c6e6b26a3c447", 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2520-g69e2cd73 (69e2cd73c52284dfd5096c66b60c6e6b26a3c447) squid (dev)", 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "entity_id": "3", 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "hostname": "smithi203", 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "pid": "43997", 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: "root": "/" 2024-03-29T04:09:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_count": 0, 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_pinned_count": 0, 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "id": 4630, 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "inst": { 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "name": { 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "client", 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "num": 4630 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.2:0", 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 1715504443 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.2:0", 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 1715504443 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "inst_str": "client.4630 192.168.0.2:0/1715504443", 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "addr_str": "192.168.0.2:0/1715504443", 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "inode_count": 1, 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_epoch": 7, 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch": 21, 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch_barrier": 0, 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "blocklisted": false, 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout: "fs_name": "cephfs" 2024-03-29T04:09:17.004 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-29T04:09:17.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.010 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-03-29T04:09:17.010 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:17.025 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-03-29T04:09:17.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:17.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.026 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:17.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.095 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-03-29T04:09:17.095 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:17.150 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-03-29T04:09:17.150 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:17.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.151 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:17.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.218 DEBUG:teuthology.orchestra.run.smithi203:> stat --file-system '--printf=%T 2024-03-29T04:09:17.218 DEBUG:teuthology.orchestra.run.smithi203:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:17.234 INFO:teuthology.orchestra.run.smithi203.stdout:fuseblk 2024-03-29T04:09:17.234 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:17.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:17.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.303 DEBUG:teuthology.orchestra.run.smithi203:> stat --file-system '--printf=%T 2024-03-29T04:09:17.303 DEBUG:teuthology.orchestra.run.smithi203:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:17.358 INFO:teuthology.orchestra.run.smithi203.stdout:fuseblk 2024-03-29T04:09:17.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:17.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:17.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:17.426 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-29T04:09:17.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:17.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.579+0000 7f208bc4a640 1 Processor -- start 2024-03-29T04:09:17.559 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.579+0000 7f208bc4a640 1 -- start start 2024-03-29T04:09:17.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f208bc4a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20840d42c0 0x7f20840d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f208bc4a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20840d4c80 con 0x7f208415e730 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f20899e6640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f208415e730 0x7f20840ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47102/0 (socket says 172.21.15.156:47102) 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f208bc4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20840cfa20 con 0x7f208406cbb0 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f20899e6640 1 -- 172.21.15.156:0/3906475803 learned_addr learned my addr 172.21.15.156:0/3906475803 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f208bc4a640 1 -- 172.21.15.156:0/3906475803 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f208406c720 con 0x7f20840d42c0 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.580+0000 7f20891e5640 1 --2- 172.21.15.156:0/3906475803 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20840d42c0 0x7f20840d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.581+0000 7f20889e4640 1 -- 172.21.15.156:0/3906475803 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942472217 0 0) 0x7f20840cfa20 con 0x7f208406cbb0 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.581+0000 7f20889e4640 1 -- 172.21.15.156:0/3906475803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2070003520 con 0x7f208406cbb0 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.581+0000 7f20889e4640 1 -- 172.21.15.156:0/3906475803 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 858448912 0 0) 0x7f20840d4c80 con 0x7f208415e730 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.581+0000 7f20891e5640 1 -- 172.21.15.156:0/3906475803 >> v1:172.21.15.203:6789/0 conn(0x7f208406cbb0 legacy=0x7f208406d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.581+0000 7f20891e5640 1 -- 172.21.15.156:0/3906475803 >> v1:172.21.15.156:6789/0 conn(0x7f208415e730 legacy=0x7f20840ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.581+0000 7f20891e5640 1 -- 172.21.15.156:0/3906475803 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20840cfd00 con 0x7f20840d42c0 2024-03-29T04:09:17.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f20891e5640 1 --2- 172.21.15.156:0/3906475803 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20840d42c0 0x7f20840d4740 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20800097c0 tx=0x7f208002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0db4f965922ce33 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:17.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f20889e4640 1 -- 172.21.15.156:0/3906475803 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080035020 con 0x7f20840d42c0 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f20889e4640 1 -- 172.21.15.156:0/3906475803 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2080031890 con 0x7f20840d42c0 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f20889e4640 1 -- 172.21.15.156:0/3906475803 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080031c60 con 0x7f20840d42c0 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 -- 172.21.15.156:0/3906475803 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20840d42c0 msgr2=0x7f20840d4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3906475803 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20840d42c0 0x7f20840d4740 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20800097c0 tx=0x7f208002b930 comp rx=0 tx=0).stop 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 -- 172.21.15.156:0/3906475803 shutdown_connections 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3906475803 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20840d42c0 0x7f20840d4740 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 -- 172.21.15.156:0/3906475803 >> 172.21.15.156:0/3906475803 conn(0x7f208408bfd0 msgr2=0x7f208409d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 -- 172.21.15.156:0/3906475803 shutdown_connections 2024-03-29T04:09:17.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.582+0000 7f208bc4a640 1 -- 172.21.15.156:0/3906475803 wait complete. 2024-03-29T04:09:17.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 Processor -- start 2024-03-29T04:09:17.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 -- start start 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f208406cbb0 0x7f2084153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 0x7f2084161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f208415e730 0x7f20840cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20840d2220 con 0x7f20840d42c0 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20840d02c0 con 0x7f208415e730 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f208bc4a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f208415eb10 con 0x7f208406cbb0 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f20891e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 0x7f2084161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f20891e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 0x7f2084161850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43706/0 (socket says 172.21.15.156:43706) 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.584+0000 7f20891e5640 1 -- 172.21.15.156:0/3662860386 learned_addr learned my addr 172.21.15.156:0/3662860386 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20899e6640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f208406cbb0 0x7f2084153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f208a1e7640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f208415e730 0x7f20840cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20891e5640 1 -- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f208406cbb0 msgr2=0x7f2084153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20891e5640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f208406cbb0 0x7f2084153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20891e5640 1 -- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f208415e730 msgr2=0x7f20840cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20891e5640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f208415e730 0x7f20840cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20891e5640 1 -- 172.21.15.156:0/3662860386 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20840c8170 con 0x7f20840d42c0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20899e6640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f208406cbb0 0x7f2084153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f20891e5640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 0x7f2084161850 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2080009710 tx=0x7f20800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080035040 con 0x7f20840d42c0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2080030b10 con 0x7f20840d42c0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20840c8450 con 0x7f20840d42c0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080005990 con 0x7f20840d42c0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.585+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20840c8a10 con 0x7f20840d42c0 2024-03-29T04:09:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.586+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20840cf6f0 con 0x7f20840d42c0 2024-03-29T04:09:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.587+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2080038020 con 0x7f20840d42c0 2024-03-29T04:09:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.587+0000 7f207a7fc640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f206806c530 0x7f206806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.588+0000 7f20899e6640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f206806c530 0x7f206806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.588+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f20800ada90 con 0x7f20840d42c0 2024-03-29T04:09:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.588+0000 7f20899e6640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f206806c530 0x7f206806e9f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2074000f60 tx=0x7f2074007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:17.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.592+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f20840cf6f0 con 0x7f20840d42c0 2024-03-29T04:09:17.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.849+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2084068610 con 0x7f20840d42c0 2024-03-29T04:09:17.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.850+0000 7f207a7fc640 1 -- 172.21.15.156:0/3662860386 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f208002d040 con 0x7f20840d42c0 2024-03-29T04:09:17.829 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:17.830 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f206806c530 msgr2=0x7f206806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f206806c530 0x7f206806e9f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2074000f60 tx=0x7f2074007e40 comp rx=0 tx=0).stop 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 msgr2=0x7f2084161850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 0x7f2084161850 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2080009710 tx=0x7f20800046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 shutdown_connections 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f206806c530 0x7f206806e9f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f208415e730 0x7f20840cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20840d42c0 0x7f2084161850 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 --2- 172.21.15.156:0/3662860386 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f208406cbb0 0x7f2084153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 >> 172.21.15.156:0/3662860386 conn(0x7f208408bfd0 msgr2=0x7f20840d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:17.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 shutdown_connections 2024-03-29T04:09:17.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.852+0000 7f208bc4a640 1 -- 172.21.15.156:0/3662860386 wait complete. 2024-03-29T04:09:17.843 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:17.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:17.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 Processor -- start 2024-03-29T04:09:17.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 -- start start 2024-03-29T04:09:17.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4e280d2a00 0x7f4e280cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e280cfd60 0x7f4e2806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e280d2fd0 con 0x7f4e2815e540 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f4e2806c970 con 0x7f4e280d2a00 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.970+0000 7f4e2da55640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4e280d32b0 con 0x7f4e280cfd60 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e1e7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4e280d2a00 0x7f4e280cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e26ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4e2815e540 0x7f4e280cf110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47114/0 (socket says 172.21.15.156:47114) 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e26ffd640 1 -- 172.21.15.156:0/3525019634 learned_addr learned my addr 172.21.15.156:0/3525019634 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e277fe640 1 --2- 172.21.15.156:0/3525019634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e280cfd60 0x7f4e2806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/3525019634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e280cfd60 msgr2=0x7f4e2806ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e1e7fc640 1 --2- 172.21.15.156:0/3525019634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e280cfd60 0x7f4e2806ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/3525019634 >> v1:172.21.15.156:6789/0 conn(0x7f4e2815e540 legacy=0x7f4e280cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/3525019634 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2806c600 con 0x7f4e280d2a00 2024-03-29T04:09:17.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.971+0000 7f4e267fc640 1 -- 172.21.15.156:0/3525019634 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489639507 0 0) 0x7f4e280d2fd0 con 0x7f4e2815e540 2024-03-29T04:09:17.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.972+0000 7f4e1e7fc640 1 --2- 172.21.15.156:0/3525019634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e280cf820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4e100097a0 tx=0x7f4e1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e19c6d0844ac2e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.972+0000 7f4e267fc640 1 -- 172.21.15.156:0/3525019634 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10035020 con 0x7f4e280d2a00 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.972+0000 7f4e267fc640 1 -- 172.21.15.156:0/3525019634 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e10031830 con 0x7f4e280d2a00 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.972+0000 7f4e267fc640 1 -- 172.21.15.156:0/3525019634 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10031c00 con 0x7f4e280d2a00 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 -- 172.21.15.156:0/3525019634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 msgr2=0x7f4e280cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 --2- 172.21.15.156:0/3525019634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e280cf820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4e100097a0 tx=0x7f4e1002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 -- 172.21.15.156:0/3525019634 shutdown_connections 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 --2- 172.21.15.156:0/3525019634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e280cfd60 0x7f4e2806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 --2- 172.21.15.156:0/3525019634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e280cf820 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 -- 172.21.15.156:0/3525019634 >> 172.21.15.156:0/3525019634 conn(0x7f4e280d4450 msgr2=0x7f4e280d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 -- 172.21.15.156:0/3525019634 shutdown_connections 2024-03-29T04:09:17.953 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.973+0000 7f4e2da55640 1 -- 172.21.15.156:0/3525019634 wait complete. 2024-03-29T04:09:17.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.974+0000 7f4e2da55640 1 Processor -- start 2024-03-29T04:09:17.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.974+0000 7f4e2da55640 1 -- start start 2024-03-29T04:09:17.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e2da55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e280cfd60 0x7f4e280cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e2da55640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e2815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e2da55640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e2815e540 0x7f4e2814dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e2da55640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2815eaa0 con 0x7f4e280d2a00 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e2da55640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2815e920 con 0x7f4e280cfd60 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e2da55640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2806d340 con 0x7f4e2815e540 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e2815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e2815ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43730/0 (socket says 172.21.15.156:43730) 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/435301276 learned_addr learned my addr 172.21.15.156:0/435301276 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e26ffd640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e280cfd60 0x7f4e280cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e277fe640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e2815e540 0x7f4e2814dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e2815e540 msgr2=0x7f4e2814dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e2815e540 0x7f4e2814dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e280cfd60 msgr2=0x7f4e280cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e280cfd60 0x7f4e280cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 -- 172.21.15.156:0/435301276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e280c9a80 con 0x7f4e280d2a00 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.975+0000 7f4e1e7fc640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e2815ec20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e10000c00 tx=0x7f4e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.976+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10035050 con 0x7f4e280d2a00 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.976+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e10004230 con 0x7f4e280d2a00 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.976+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e280c9d60 con 0x7f4e280d2a00 2024-03-29T04:09:17.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.976+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1003b800 con 0x7f4e280d2a00 2024-03-29T04:09:17.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.976+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e280ca320 con 0x7f4e280d2a00 2024-03-29T04:09:17.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.977+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e280a23a0 con 0x7f4e280d2a00 2024-03-29T04:09:17.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.977+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4e10038020 con 0x7f4e280d2a00 2024-03-29T04:09:17.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.977+0000 7f4e24ff9640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4e0806c530 0x7f4e0806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:17.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.978+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4e100ad3e0 con 0x7f4e280d2a00 2024-03-29T04:09:17.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.978+0000 7f4e26ffd640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4e0806c530 0x7f4e0806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:17.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.978+0000 7f4e26ffd640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4e0806c530 0x7f4e0806e9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4e18007fd0 tx=0x7f4e18007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:17.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:17.982+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4e10040070 con 0x7f4e280d2a00 2024-03-29T04:09:18.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.161+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e2804f9d0 con 0x7f4e280d2a00 2024-03-29T04:09:18.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.162+0000 7f4e24ff9640 1 -- 172.21.15.156:0/435301276 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f4e1007b3b0 con 0x7f4e280d2a00 2024-03-29T04:09:18.142 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4e0806c530 msgr2=0x7f4e0806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4e0806c530 0x7f4e0806e9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4e18007fd0 tx=0x7f4e18007de0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 msgr2=0x7f4e2815ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e2815ec20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e10000c00 tx=0x7f4e100046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 shutdown_connections 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4e0806c530 0x7f4e0806e9f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4e2815e540 0x7f4e2814dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e280d2a00 0x7f4e2815ec20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 --2- 172.21.15.156:0/435301276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e280cfd60 0x7f4e280cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 >> 172.21.15.156:0/435301276 conn(0x7f4e280d4450 msgr2=0x7f4e280b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 shutdown_connections 2024-03-29T04:09:18.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.164+0000 7f4e2da55640 1 -- 172.21.15.156:0/435301276 wait complete. 2024-03-29T04:09:18.155 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:18.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.280+0000 7ff71478d640 1 Processor -- start 2024-03-29T04:09:18.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.280+0000 7ff71478d640 1 -- start start 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.281+0000 7ff71478d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff70c0d2120 0x7ff70c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.281+0000 7ff71478d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff70c06cb10 0x7ff70c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.281+0000 7ff71478d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff70c0d2df0 con 0x7ff70c15e920 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.281+0000 7ff71478d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff70c0cf7a0 con 0x7ff70c0d2120 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.281+0000 7ff71478d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff70c0cf190 con 0x7ff70c06cb10 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.281+0000 7ff711d28640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff70c0d2120 0x7ff70c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff712529640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff70c15e920 0x7ff70c0d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47118/0 (socket says 172.21.15.156:47118) 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff711d28640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff70c0d2120 0x7ff70c06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43742/0 (socket says 172.21.15.156:43742) 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff712529640 1 -- 172.21.15.156:0/4010985454 learned_addr learned my addr 172.21.15.156:0/4010985454 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff712d2a640 1 --2- 172.21.15.156:0/4010985454 >> v2:172.21.15.203:3300/0 conn(0x7ff70c06cb10 0x7ff70c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff711d28640 1 -- 172.21.15.156:0/4010985454 >> v2:172.21.15.203:3300/0 conn(0x7ff70c06cb10 msgr2=0x7ff70c0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff711d28640 1 --2- 172.21.15.156:0/4010985454 >> v2:172.21.15.203:3300/0 conn(0x7ff70c06cb10 0x7ff70c0cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff711d28640 1 -- 172.21.15.156:0/4010985454 >> v1:172.21.15.156:6789/0 conn(0x7ff70c15e920 legacy=0x7ff70c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff711527640 1 -- 172.21.15.156:0/4010985454 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452308212 0 0) 0x7ff70c0d2df0 con 0x7ff70c15e920 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff711d28640 1 -- 172.21.15.156:0/4010985454 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70c0d3080 con 0x7ff70c0d2120 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.282+0000 7ff712d2a640 1 --2- 172.21.15.156:0/4010985454 >> v2:172.21.15.203:3300/0 conn(0x7ff70c06cb10 0x7ff70c0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff711d28640 1 --2- 172.21.15.156:0/4010985454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c0d2120 0x7ff70c06c5d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff70c08ca00 tx=0x7ff70802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fdd4c9bd455947 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff711527640 1 -- 172.21.15.156:0/4010985454 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708035020 con 0x7ff70c0d2120 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff711527640 1 -- 172.21.15.156:0/4010985454 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff708031830 con 0x7ff70c0d2120 2024-03-29T04:09:18.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff711527640 1 -- 172.21.15.156:0/4010985454 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708031c00 con 0x7ff70c0d2120 2024-03-29T04:09:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 -- 172.21.15.156:0/4010985454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c0d2120 msgr2=0x7ff70c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 --2- 172.21.15.156:0/4010985454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c0d2120 0x7ff70c06c5d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff70c08ca00 tx=0x7ff70802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 -- 172.21.15.156:0/4010985454 shutdown_connections 2024-03-29T04:09:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 --2- 172.21.15.156:0/4010985454 >> v2:172.21.15.203:3300/0 conn(0x7ff70c06cb10 0x7ff70c0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 --2- 172.21.15.156:0/4010985454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c0d2120 0x7ff70c06c5d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 -- 172.21.15.156:0/4010985454 >> 172.21.15.156:0/4010985454 conn(0x7ff70c09d7a0 msgr2=0x7ff70c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 -- 172.21.15.156:0/4010985454 shutdown_connections 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.283+0000 7ff71478d640 1 -- 172.21.15.156:0/4010985454 wait complete. 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 Processor -- start 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 -- start start 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 0x7ff70c0c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff70c0d2120 0x7ff70c15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff70c15e920 0x7ff70c15f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff70c0d34e0 con 0x7ff70c06cb10 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff70c0d3360 con 0x7ff70c15e920 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff71478d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff70c0d0170 con 0x7ff70c0d2120 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff712529640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 0x7ff70c0c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff712529640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 0x7ff70c0c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43752/0 (socket says 172.21.15.156:43752) 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff712529640 1 -- 172.21.15.156:0/52339 learned_addr learned my addr 172.21.15.156:0/52339 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:18.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.285+0000 7ff712d2a640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff70c15e920 0x7ff70c15f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff711d28640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff70c0d2120 0x7ff70c15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff712529640 1 -- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff70c0d2120 msgr2=0x7ff70c15ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff712529640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff70c0d2120 0x7ff70c15ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff712529640 1 -- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff70c15e920 msgr2=0x7ff70c15f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff712529640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff70c15e920 0x7ff70c15f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff712529640 1 -- 172.21.15.156:0/52339 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70c0c9890 con 0x7ff70c06cb10 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff711d28640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff70c0d2120 0x7ff70c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff712529640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 0x7ff70c0c8dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c430 tx=0x7ff6fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0074e0 con 0x7ff70c06cb10 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6fc0047e0 con 0x7ff70c06cb10 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff70c0c9b70 con 0x7ff70c06cb10 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc004bd0 con 0x7ff70c06cb10 2024-03-29T04:09:18.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.286+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff70c0ca130 con 0x7ff70c06cb10 2024-03-29T04:09:18.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.287+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff70c0a2480 con 0x7ff70c06cb10 2024-03-29T04:09:18.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.287+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff6fc011070 con 0x7ff70c06cb10 2024-03-29T04:09:18.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.288+0000 7ff702ffd640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff6f006c530 0x7ff6f006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.288+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff6fc08b2e0 con 0x7ff70c06cb10 2024-03-29T04:09:18.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.288+0000 7ff711d28640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff6f006c530 0x7ff6f006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.288+0000 7ff711d28640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff6f006c530 0x7ff6f006e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff708036000 tx=0x7ff708008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.292+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff6fc059900 con 0x7ff70c06cb10 2024-03-29T04:09:18.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.548+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff70c0cfdd0 con 0x7ff70c06cb10 2024-03-29T04:09:18.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.549+0000 7ff702ffd640 1 -- 172.21.15.156:0/52339 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7ff6fc059200 con 0x7ff70c06cb10 2024-03-29T04:09:18.528 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:18.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.550+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff6f006c530 msgr2=0x7ff6f006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.550+0000 7ff71478d640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff6f006c530 0x7ff6f006e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff708036000 tx=0x7ff708008030 comp rx=0 tx=0).stop 2024-03-29T04:09:18.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.550+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 msgr2=0x7ff70c0c8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.550+0000 7ff71478d640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 0x7ff70c0c8dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c430 tx=0x7ff6fc00c8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 shutdown_connections 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff6f006c530 0x7ff6f006e9f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff70c15e920 0x7ff70c15f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff70c0d2120 0x7ff70c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 --2- 172.21.15.156:0/52339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff70c06cb10 0x7ff70c0c8dd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 >> 172.21.15.156:0/52339 conn(0x7ff70c09d7a0 msgr2=0x7ff70c0b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 shutdown_connections 2024-03-29T04:09:18.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.551+0000 7ff71478d640 1 -- 172.21.15.156:0/52339 wait complete. 2024-03-29T04:09:18.541 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:18.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.667+0000 7f2ea638d640 1 Processor -- start 2024-03-29T04:09:18.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.667+0000 7f2ea638d640 1 -- start start 2024-03-29T04:09:18.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2ea638d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 0x7f2ea00cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2ea638d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea00d2fd0 con 0x7f2ea00cfd60 2024-03-29T04:09:18.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2ea638d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea00d32b0 con 0x7f2ea015e540 2024-03-29T04:09:18.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2ea638d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea006c970 con 0x7f2ea00d2a00 2024-03-29T04:09:18.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2ea492a640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2ea00cfd60 0x7f2ea006ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47134/0 (socket says 172.21.15.156:47134) 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2ea492a640 1 -- 172.21.15.156:0/2774750510 learned_addr learned my addr 172.21.15.156:0/2774750510 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9f7fe640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 0x7f2ea00cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216931651 0 0) 0x7f2ea00d2fd0 con 0x7f2ea00cfd60 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e88003520 con 0x7f2ea00cfd60 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2089189166 0 0) 0x7f2e88003520 con 0x7f2ea00cfd60 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 msgr2=0x7f2ea00cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 0x7f2ea00cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 >> v1:172.21.15.203:6789/0 conn(0x7f2ea015e540 legacy=0x7f2ea00cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea006c600 con 0x7f2ea00cfd60 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.668+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f2e980019f0 con 0x7f2ea00cfd60 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 >> v1:172.21.15.156:6789/0 conn(0x7f2ea00cfd60 legacy=0x7f2ea006ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 0x7f2e880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e88003f40 0x7f2e88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2e880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea006c970 con 0x7f2e88004d40 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8800ba70 con 0x7f2e88003f40 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8800bc40 con 0x7f2ea00d2a00 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2ea492a640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2e880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9f7fe640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e88003f40 0x7f2e88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9ffff640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 0x7f2e880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2ea492a640 1 -- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 msgr2=0x7f2e880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2ea492a640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00d2a00 0x7f2e880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2ea492a640 1 -- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e88003f40 msgr2=0x7f2e88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2ea492a640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e88003f40 0x7f2e88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2ea492a640 1 -- 172.21.15.156:0/2774750510 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e88003520 con 0x7f2e88004d40 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.669+0000 7f2e9f7fe640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e88003f40 0x7f2e88004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea492a640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2e880051c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800a860 tx=0x7f2e88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c52d373123d71f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9800a480 con 0x7f2e88004d40 2024-03-29T04:09:18.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e98034630 con 0x7f2e88004d40 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2e9effd640 1 -- 172.21.15.156:0/2774750510 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98034a80 con 0x7f2e88004d40 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea638d640 1 -- 172.21.15.156:0/2774750510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 msgr2=0x7f2e880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea638d640 1 --2- 172.21.15.156:0/2774750510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2e880051c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800a860 tx=0x7f2e88005830 comp rx=0 tx=0).stop 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea492a640 1 -- 172.21.15.156:0/2774750510 reap_dead start 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea638d640 1 -- 172.21.15.156:0/2774750510 shutdown_connections 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea638d640 1 -- 172.21.15.156:0/2774750510 >> 172.21.15.156:0/2774750510 conn(0x7f2ea00d4450 msgr2=0x7f2ea00d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea638d640 1 -- 172.21.15.156:0/2774750510 shutdown_connections 2024-03-29T04:09:18.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.670+0000 7f2ea638d640 1 -- 172.21.15.156:0/2774750510 wait complete. 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.672+0000 7f2ea638d640 1 Processor -- start 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.672+0000 7f2ea638d640 1 -- start start 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.672+0000 7f2ea638d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2ea0151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00b9c40 0x7f2ea015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ea00b4ee0 0x7f2ea00b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea015eaa0 con 0x7f2e88004d40 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea015e920 con 0x7f2ea00b4ee0 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea00cf820 con 0x7f2ea00b9c40 2024-03-29T04:09:18.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2ea0151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2ea0151790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43774/0 (socket says 172.21.15.156:43774) 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 -- 172.21.15.156:0/3882058214 learned_addr learned my addr 172.21.15.156:0/3882058214 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9f7fe640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00b9c40 0x7f2ea015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea492a640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ea00b4ee0 0x7f2ea00b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 -- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00b9c40 msgr2=0x7f2ea015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00b9c40 0x7f2ea015ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 -- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ea00b4ee0 msgr2=0x7f2ea00b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ea00b4ee0 0x7f2ea00b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 -- 172.21.15.156:0/3882058214 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea0161bf0 con 0x7f2e88004d40 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea492a640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ea00b4ee0 0x7f2ea00b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9ffff640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2ea0151790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400f7f0 tx=0x7f2e9400fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400fea0 con 0x7f2e88004d40 2024-03-29T04:09:18.657 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e94018070 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea0161350 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400d6d0 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.673+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea0161910 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.674+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea00a23a0 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.675+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2e9400d870 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.676+0000 7f2e9cff9640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e8c06c530 0x7f2e8c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.676+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2e9408b570 con 0x7f2e88004d40 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.676+0000 7f2e9f7fe640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e8c06c530 0x7f2e8c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.658 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.676+0000 7f2e9f7fe640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e8c06c530 0x7f2e8c06e9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ea015b2e0 tx=0x7f2e90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.680+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2e94059c20 con 0x7f2e88004d40 2024-03-29T04:09:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.859+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2ea004f9d0 con 0x7f2e88004d40 2024-03-29T04:09:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.859+0000 7f2e9cff9640 1 -- 172.21.15.156:0/3882058214 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f2e94059520 con 0x7f2e88004d40 2024-03-29T04:09:18.840 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e8c06c530 msgr2=0x7f2e8c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e8c06c530 0x7f2e8c06e9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ea015b2e0 tx=0x7f2e90008040 comp rx=0 tx=0).stop 2024-03-29T04:09:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 msgr2=0x7f2ea0151790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2ea0151790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400f7f0 tx=0x7f2e9400fcb0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 shutdown_connections 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e8c06c530 0x7f2e8c06e9f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ea00b4ee0 0x7f2ea00b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ea00b9c40 0x7f2ea015ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 --2- 172.21.15.156:0/3882058214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e88004d40 0x7f2ea0151790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.861+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 >> 172.21.15.156:0/3882058214 conn(0x7f2ea00d4450 msgr2=0x7f2ea006dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.862+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 shutdown_connections 2024-03-29T04:09:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.862+0000 7f2ea638d640 1 -- 172.21.15.156:0/3882058214 wait complete. 2024-03-29T04:09:18.852 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:18.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba39f31640 1 Processor -- start 2024-03-29T04:09:18.954 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba39f31640 1 -- start start 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba39f31640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba3415e730 0x7fba340ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba39f31640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba340cfa20 con 0x7fba340cf3e0 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba39f31640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba340cfd00 con 0x7fba340d49d0 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba39f31640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fba3406c720 con 0x7fba3415e730 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba32ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fba340cf3e0 0x7fba340d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47144/0 (socket says 172.21.15.156:47144) 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.975+0000 7fba32ffd640 1 -- 172.21.15.156:0/2467505485 learned_addr learned my addr 172.21.15.156:0/2467505485 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963033347 0 0) 0x7fba340cfa20 con 0x7fba340cf3e0 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba1c003520 con 0x7fba340cf3e0 2024-03-29T04:09:18.958 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 --2- 172.21.15.156:0/2467505485 >> v2:172.21.15.203:3300/0 conn(0x7fba3415e730 0x7fba340ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1858921458 0 0) 0x7fba1c003520 con 0x7fba340cf3e0 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 >> v2:172.21.15.203:3300/0 conn(0x7fba3415e730 msgr2=0x7fba340ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 --2- 172.21.15.156:0/2467505485 >> v2:172.21.15.203:3300/0 conn(0x7fba3415e730 0x7fba340ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 >> v1:172.21.15.203:6789/0 conn(0x7fba340d49d0 legacy=0x7fba3406cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba340cffe0 con 0x7fba340cf3e0 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fba280017f0 con 0x7fba340cf3e0 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 >> v1:172.21.15.156:6789/0 conn(0x7fba340cf3e0 legacy=0x7fba340d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba1c0039b0 0x7fba1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c004800 0x7fba1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba1c0051c0 0x7fba1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba3406c720 con 0x7fba1c0039b0 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c00c160 con 0x7fba1c004800 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fba1c00c360 con 0x7fba1c0051c0 2024-03-29T04:09:18.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba1c0039b0 0x7fba1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba32ffd640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c004800 0x7fba1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba33fff640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba1c0051c0 0x7fba1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 -- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba1c0051c0 msgr2=0x7fba1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba1c0051c0 0x7fba1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 -- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c004800 msgr2=0x7fba1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:18.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c004800 0x7fba1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba337fe640 1 -- 172.21.15.156:0/2467505485 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c003520 con 0x7fba1c0039b0 2024-03-29T04:09:18.961 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.976+0000 7fba33fff640 1 -- 172.21.15.156:0/2467505485 reap_dead start 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.977+0000 7fba337fe640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba1c0039b0 0x7fba1c003dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fba24009ac0 tx=0x7fba1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5623edd14f3e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.977+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24039070 con 0x7fba1c0039b0 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.977+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba2402c6d0 con 0x7fba1c0039b0 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.977+0000 7fba327fc640 1 -- 172.21.15.156:0/2467505485 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24009690 con 0x7fba1c0039b0 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 -- 172.21.15.156:0/2467505485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba1c0039b0 msgr2=0x7fba1c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba1c0039b0 0x7fba1c003dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fba24009ac0 tx=0x7fba1c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 -- 172.21.15.156:0/2467505485 shutdown_connections 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 --2- 172.21.15.156:0/2467505485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba1c0039b0 0x7fba1c003dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 -- 172.21.15.156:0/2467505485 >> 172.21.15.156:0/2467505485 conn(0x7fba3408bfd0 msgr2=0x7fba3409d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 -- 172.21.15.156:0/2467505485 shutdown_connections 2024-03-29T04:09:18.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.978+0000 7fba39f31640 1 -- 172.21.15.156:0/2467505485 wait complete. 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.979+0000 7fba39f31640 1 Processor -- start 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.979+0000 7fba39f31640 1 -- start start 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba39f31640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0039b0 0x7fba341f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba39f31640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 0x7fba3415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba39f31640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba341503c0 0x7fba34150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba39f31640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba3415eb10 con 0x7fba34153000 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba39f31640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba340d2220 con 0x7fba1c0039b0 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba39f31640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fba3415ec90 con 0x7fba341503c0 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 0x7fba3415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 0x7fba3415ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43794/0 (socket says 172.21.15.156:43794) 2024-03-29T04:09:18.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 -- 172.21.15.156:0/977928639 learned_addr learned my addr 172.21.15.156:0/977928639 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba33fff640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba341503c0 0x7fba34150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba337fe640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0039b0 0x7fba341f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 -- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba341503c0 msgr2=0x7fba34150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba341503c0 0x7fba34150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 -- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0039b0 msgr2=0x7fba341f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0039b0 0x7fba341f3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.980+0000 7fba32ffd640 1 -- 172.21.15.156:0/977928639 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba340c9250 con 0x7fba34153000 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.981+0000 7fba32ffd640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 0x7fba3415ee10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fba2800f800 tx=0x7fba2800fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.981+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba280072e0 con 0x7fba34153000 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.981+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba28018070 con 0x7fba34153000 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.981+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba340c9530 con 0x7fba34153000 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.981+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba280057a0 con 0x7fba34153000 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.981+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba340c9af0 con 0x7fba34153000 2024-03-29T04:09:18.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.982+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba34068610 con 0x7fba34153000 2024-03-29T04:09:18.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.982+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fba28005940 con 0x7fba34153000 2024-03-29T04:09:18.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.983+0000 7fba1bfff640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fba2006c560 0x7fba2006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:18.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.983+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fba2808b4e0 con 0x7fba34153000 2024-03-29T04:09:18.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.983+0000 7fba337fe640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fba2006c560 0x7fba2006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:18.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.983+0000 7fba337fe640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fba2006c560 0x7fba2006ea20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba24036000 tx=0x7fba24030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:18.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:18.990+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fba28059570 con 0x7fba34153000 2024-03-29T04:09:19.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.166+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba3406b5b0 con 0x7fba34153000 2024-03-29T04:09:19.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.168+0000 7fba1bfff640 1 -- 172.21.15.156:0/977928639 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fba2805d870 con 0x7fba34153000 2024-03-29T04:09:19.148 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:19.148 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:19.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.169+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fba2006c560 msgr2=0x7fba2006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.169+0000 7fba39f31640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fba2006c560 0x7fba2006ea20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba24036000 tx=0x7fba24030000 comp rx=0 tx=0).stop 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.169+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 msgr2=0x7fba3415ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.169+0000 7fba39f31640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 0x7fba3415ee10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fba2800f800 tx=0x7fba2800fcc0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 shutdown_connections 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fba2006c560 0x7fba2006ea20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba341503c0 0x7fba34150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba34153000 0x7fba3415ee10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 --2- 172.21.15.156:0/977928639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0039b0 0x7fba341f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 >> 172.21.15.156:0/977928639 conn(0x7fba3408bfd0 msgr2=0x7fba340d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 shutdown_connections 2024-03-29T04:09:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.170+0000 7fba39f31640 1 -- 172.21.15.156:0/977928639 wait complete. 2024-03-29T04:09:19.162 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:19.162 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:19.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.287+0000 7fa6410d5640 1 Processor -- start 2024-03-29T04:09:19.267 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.287+0000 7fa6410d5640 1 -- start start 2024-03-29T04:09:19.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa6410d5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa63c145c70 0x7fa63c0b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa6410d5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c0cf510 0x7fa63c0d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa6410d5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63c15cb10 con 0x7fa63c0b4310 2024-03-29T04:09:19.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa6410d5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa63c0cf980 con 0x7fa63c145c70 2024-03-29T04:09:19.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa6410d5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa63c15cdf0 con 0x7fa63c0cf510 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa63b59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c0cf510 0x7fa63c0d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa63ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa63c145c70 0x7fa63c0b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa63ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa63c145c70 0x7fa63c0b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:54914/0 (socket says 172.21.15.156:54914) 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa63b59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c0cf510 0x7fa63c0d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:44824/0 (socket says 172.21.15.156:44824) 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa63a59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa63c0b4310 0x7fa63c0cee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:60022/0 (socket says 172.21.15.156:60022) 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.288+0000 7fa63ad9d640 1 -- 172.21.15.156:0/1593425125 learned_addr learned my addr 172.21.15.156:0/1593425125 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.289+0000 7fa63ad9d640 1 -- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c0cf510 msgr2=0x7fa63c0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.289+0000 7fa63ad9d640 1 --2- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c0cf510 0x7fa63c0d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.289+0000 7fa63ad9d640 1 -- 172.21.15.156:0/1593425125 >> v1:172.21.15.203:6789/0 conn(0x7fa63c0b4310 legacy=0x7fa63c0cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.289+0000 7fa63ad9d640 1 -- 172.21.15.156:0/1593425125 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c15d0d0 con 0x7fa63c145c70 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa63ad9d640 1 --2- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c145c70 0x7fa63c0b3dd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009870 tx=0x7fa62c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0676599e6e4f53f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa639d9b640 1 -- 172.21.15.156:0/1593425125 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c035020 con 0x7fa63c145c70 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa639d9b640 1 -- 172.21.15.156:0/1593425125 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62c031640 con 0x7fa63c145c70 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa639d9b640 1 -- 172.21.15.156:0/1593425125 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c02d040 con 0x7fa63c145c70 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 -- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c145c70 msgr2=0x7fa63c0b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 --2- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c145c70 0x7fa63c0b3dd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009870 tx=0x7fa62c02b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 -- 172.21.15.156:0/1593425125 shutdown_connections 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 --2- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c0cf510 0x7fa63c0d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 --2- 172.21.15.156:0/1593425125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c145c70 0x7fa63c0b3dd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 -- 172.21.15.156:0/1593425125 >> 172.21.15.156:0/1593425125 conn(0x7fa63c0b7660 msgr2=0x7fa63c0b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 -- 172.21.15.156:0/1593425125 shutdown_connections 2024-03-29T04:09:19.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.290+0000 7fa6410d5640 1 -- 172.21.15.156:0/1593425125 wait complete. 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 Processor -- start 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 -- start start 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c0b4310 0x7fa63c1f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 0x7fa63c06c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c145c70 0x7fa63c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c1461d0 con 0x7fa63c0cf510 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c146050 con 0x7fa63c0b4310 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.292+0000 7fa6410d5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa63c0d1430 con 0x7fa63c145c70 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 0x7fa63c06c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 0x7fa63c06c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43798/0 (socket says 172.21.15.156:43798) 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 -- 172.21.15.156:0/887410845 learned_addr learned my addr 172.21.15.156:0/887410845 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63ad9d640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c0b4310 0x7fa63c1f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63b59e640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c145c70 0x7fa63c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 -- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c145c70 msgr2=0x7fa63c06fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c145c70 0x7fa63c06fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 -- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c0b4310 msgr2=0x7fa63c1f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c0b4310 0x7fa63c1f4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 -- 172.21.15.156:0/887410845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c153cc0 con 0x7fa63c0cf510 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63ad9d640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c0b4310 0x7fa63c1f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa63a59c640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 0x7fa63c06c700 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa63000c400 tx=0x7fa63000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6300074e0 con 0x7fa63c0cf510 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c1f5270 con 0x7fa63c0cf510 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa630007680 con 0x7fa63c0cf510 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630004a00 con 0x7fa63c0cf510 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.293+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c1f5830 con 0x7fa63c0cf510 2024-03-29T04:09:19.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.294+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c15c800 con 0x7fa63c0cf510 2024-03-29T04:09:19.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.295+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa63001a020 con 0x7fa63c0cf510 2024-03-29T04:09:19.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.295+0000 7fa6237fe640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa62806c530 0x7fa62806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.295+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa63008ae30 con 0x7fa63c0cf510 2024-03-29T04:09:19.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.295+0000 7fa63ad9d640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa62806c530 0x7fa62806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.296+0000 7fa63ad9d640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa62806c530 0x7fa62806e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.300+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa63c15c800 con 0x7fa63c0cf510 2024-03-29T04:09:19.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.503+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa63c04f9d0 con 0x7fa63c0cf510 2024-03-29T04:09:19.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.503+0000 7fa6237fe640 1 -- 172.21.15.156:0/887410845 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fa630058d80 con 0x7fa63c0cf510 2024-03-29T04:09:19.484 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa62806c530 msgr2=0x7fa62806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa62806c530 0x7fa62806e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c03c040 comp rx=0 tx=0).stop 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 msgr2=0x7fa63c06c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 0x7fa63c06c700 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa63000c400 tx=0x7fa63000c8c0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 shutdown_connections 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa62806c530 0x7fa62806e9f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa63c145c70 0x7fa63c06fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa63c0cf510 0x7fa63c06c700 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 --2- 172.21.15.156:0/887410845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa63c0b4310 0x7fa63c1f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 >> 172.21.15.156:0/887410845 conn(0x7fa63c0b7660 msgr2=0x7fa63c15a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:19.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 shutdown_connections 2024-03-29T04:09:19.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.505+0000 7fa6410d5640 1 -- 172.21.15.156:0/887410845 wait complete. 2024-03-29T04:09:19.496 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.621+0000 7f45ac6d9640 1 Processor -- start 2024-03-29T04:09:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.621+0000 7f45ac6d9640 1 -- start start 2024-03-29T04:09:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.622+0000 7f45ac6d9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45a409aa40 0x7f45a409ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.622+0000 7f45ac6d9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a406cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.622+0000 7f45ac6d9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a409b360 con 0x7f45a406c6c0 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.622+0000 7f45ac6d9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f45a40b4c60 con 0x7f45a409aa40 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.622+0000 7f45ac6d9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f45a4160000 con 0x7f45a415f8e0 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45a409aa40 0x7f45a409ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45a409aa40 0x7f45a409ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43814/0 (socket says 172.21.15.156:43814) 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 -- 172.21.15.156:0/1480421780 learned_addr learned my addr 172.21.15.156:0/1480421780 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aac76640 1 --2- 172.21.15.156:0/1480421780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a406cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 -- 172.21.15.156:0/1480421780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 msgr2=0x7f45a406cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 --2- 172.21.15.156:0/1480421780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a406cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 -- 172.21.15.156:0/1480421780 >> v1:172.21.15.203:6789/0 conn(0x7f45a406c6c0 legacy=0x7f45a40d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.623+0000 7f45aa475640 1 -- 172.21.15.156:0/1480421780 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a4162430 con 0x7f45a409aa40 2024-03-29T04:09:19.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.624+0000 7f45aa475640 1 --2- 172.21.15.156:0/1480421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a409aa40 0x7f45a409ae20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92952ff2b6e8e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.624+0000 7f45a9473640 1 -- 172.21.15.156:0/1480421780 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c035020 con 0x7f45a409aa40 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.624+0000 7f45a9473640 1 -- 172.21.15.156:0/1480421780 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f459c031640 con 0x7f45a409aa40 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.624+0000 7f45a9473640 1 -- 172.21.15.156:0/1480421780 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c02d040 con 0x7f45a409aa40 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1480421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a409aa40 msgr2=0x7f45a409ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1480421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a409aa40 0x7f45a409ae20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c02b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1480421780 shutdown_connections 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1480421780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a406cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1480421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a409aa40 0x7f45a409ae20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1480421780 >> 172.21.15.156:0/1480421780 conn(0x7f45a40b3df0 msgr2=0x7f45a40b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1480421780 shutdown_connections 2024-03-29T04:09:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.625+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1480421780 wait complete. 2024-03-29T04:09:19.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.627+0000 7f45ac6d9640 1 Processor -- start 2024-03-29T04:09:19.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.627+0000 7f45ac6d9640 1 -- start start 2024-03-29T04:09:19.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45ac6d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 0x7f45a4156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45ac6d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45a409aa40 0x7f45a4157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45ac6d9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a415cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45ac6d9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41629f0 con 0x7f45a406c6c0 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45ac6d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4163040 con 0x7f45a409aa40 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45ac6d9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f45a4162cd0 con 0x7f45a415f8e0 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 0x7f45a4156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 0x7f45a4156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43830/0 (socket says 172.21.15.156:43830) 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 -- 172.21.15.156:0/1133469078 learned_addr learned my addr 172.21.15.156:0/1133469078 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45a9c74640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45a409aa40 0x7f45a4157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aac76640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a415cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 -- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 msgr2=0x7f45a415cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a415cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 -- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45a409aa40 msgr2=0x7f45a4157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45a409aa40 0x7f45a4157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 -- 172.21.15.156:0/1133469078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a40c6c20 con 0x7f45a406c6c0 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.628+0000 7f45aa475640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 0x7f45a4156620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f459c02b930 tx=0x7f459c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.629+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c040070 con 0x7f45a406c6c0 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.629+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f459c039c10 con 0x7f45a406c6c0 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.629+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c035040 con 0x7f45a406c6c0 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.629+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a40c6f00 con 0x7f45a406c6c0 2024-03-29T04:09:19.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.629+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a40c74c0 con 0x7f45a406c6c0 2024-03-29T04:09:19.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.630+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a40b7e30 con 0x7f45a406c6c0 2024-03-29T04:09:19.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.630+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f459c04b060 con 0x7f45a406c6c0 2024-03-29T04:09:19.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.630+0000 7f45a2ffd640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f459006c560 0x7f459006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.631+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f459c0ad270 con 0x7f45a406c6c0 2024-03-29T04:09:19.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.631+0000 7f45a9c74640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f459006c560 0x7f459006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.631+0000 7f45a9c74640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f459006c560 0x7f459006ea20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4598006fd0 tx=0x7f4598008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.635+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f45a40b7e30 con 0x7f45a406c6c0 2024-03-29T04:09:19.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.811+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45a40a0450 con 0x7f45a406c6c0 2024-03-29T04:09:19.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.811+0000 7f45a2ffd640 1 -- 172.21.15.156:0/1133469078 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f459c02f300 con 0x7f45a406c6c0 2024-03-29T04:09:19.792 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f459006c560 msgr2=0x7f459006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f459006c560 0x7f459006ea20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4598006fd0 tx=0x7f4598008040 comp rx=0 tx=0).stop 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 msgr2=0x7f45a4156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 0x7f45a4156620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f459c02b930 tx=0x7f459c0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 shutdown_connections 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f459006c560 0x7f459006ea20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45a415f8e0 0x7f45a415cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45a409aa40 0x7f45a4157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 --2- 172.21.15.156:0/1133469078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45a406c6c0 0x7f45a4156620 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 >> 172.21.15.156:0/1133469078 conn(0x7f45a40b3df0 msgr2=0x7f45a4161c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 shutdown_connections 2024-03-29T04:09:19.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.814+0000 7f45ac6d9640 1 -- 172.21.15.156:0/1133469078 wait complete. 2024-03-29T04:09:19.809 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:19.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.935+0000 7f63db982640 1 Processor -- start 2024-03-29T04:09:19.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.935+0000 7f63db982640 1 -- start start 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63db982640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f63d4071020 0x7f63d4071400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63db982640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f63d4141ca0 0x7f63d415c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63db982640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d4071940 con 0x7f63d415ceb0 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63db982640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f63d415d960 con 0x7f63d4141ca0 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63db982640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f63d413f7f0 con 0x7f63d4071020 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63d8f1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f63d4141ca0 0x7f63d415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63d8f1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f63d4141ca0 0x7f63d415c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43832/0 (socket says 172.21.15.156:43832) 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63d8f1d640 1 -- 172.21.15.156:0/1481002250 learned_addr learned my addr 172.21.15.156:0/1481002250 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63d971e640 1 --2- 172.21.15.156:0/1481002250 >> v2:172.21.15.203:3300/0 conn(0x7f63d4071020 0x7f63d4071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63d8f1d640 1 -- 172.21.15.156:0/1481002250 >> v2:172.21.15.203:3300/0 conn(0x7f63d4071020 msgr2=0x7f63d4071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.936+0000 7f63d8f1d640 1 --2- 172.21.15.156:0/1481002250 >> v2:172.21.15.203:3300/0 conn(0x7f63d4071020 0x7f63d4071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.937+0000 7f63d8f1d640 1 -- 172.21.15.156:0/1481002250 >> v1:172.21.15.203:6789/0 conn(0x7f63d415ceb0 legacy=0x7f63d415ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.937+0000 7f63d8f1d640 1 -- 172.21.15.156:0/1481002250 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d415d3a0 con 0x7f63d4141ca0 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.937+0000 7f63d8f1d640 1 --2- 172.21.15.156:0/1481002250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d4141ca0 0x7f63d415c970 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f63d406c560 tx=0x7f63c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df160aca43e8335b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.937+0000 7f63cbfff640 1 -- 172.21.15.156:0/1481002250 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4034070 con 0x7f63d4141ca0 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.937+0000 7f63cbfff640 1 -- 172.21.15.156:0/1481002250 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c4030860 con 0x7f63d4141ca0 2024-03-29T04:09:19.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.937+0000 7f63cbfff640 1 -- 172.21.15.156:0/1481002250 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4030c30 con 0x7f63d4141ca0 2024-03-29T04:09:19.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 -- 172.21.15.156:0/1481002250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d4141ca0 msgr2=0x7f63d415c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 --2- 172.21.15.156:0/1481002250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d4141ca0 0x7f63d415c970 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f63d406c560 tx=0x7f63c402b9a0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 -- 172.21.15.156:0/1481002250 shutdown_connections 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 --2- 172.21.15.156:0/1481002250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d4141ca0 0x7f63d415c970 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 --2- 172.21.15.156:0/1481002250 >> v2:172.21.15.203:3300/0 conn(0x7f63d4071020 0x7f63d4071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 -- 172.21.15.156:0/1481002250 >> 172.21.15.156:0/1481002250 conn(0x7f63d40b3eb0 msgr2=0x7f63d40b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 -- 172.21.15.156:0/1481002250 shutdown_connections 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.938+0000 7f63db982640 1 -- 172.21.15.156:0/1481002250 wait complete. 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.939+0000 7f63db982640 1 Processor -- start 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.939+0000 7f63db982640 1 -- start start 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63db982640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63d4071020 0x7f63d4140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63db982640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63d4141ca0 0x7f63d4144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63db982640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 0x7f63d4145150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63db982640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f63d40d1280 con 0x7f63d415ceb0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63db982640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f63d40d0fa0 con 0x7f63d4071020 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63db982640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f63d40d27b0 con 0x7f63d4141ca0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 0x7f63d4145150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 0x7f63d4145150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43842/0 (socket says 172.21.15.156:43842) 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 -- 172.21.15.156:0/3135218680 learned_addr learned my addr 172.21.15.156:0/3135218680 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d971e640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63d4071020 0x7f63d4140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d8f1d640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63d4141ca0 0x7f63d4144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 -- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63d4141ca0 msgr2=0x7f63d4144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63d4141ca0 0x7f63d4144c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 -- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63d4071020 msgr2=0x7f63d4140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63d4071020 0x7f63d4140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.940+0000 7f63d9f1f640 1 -- 172.21.15.156:0/3135218680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d4143760 con 0x7f63d415ceb0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.941+0000 7f63d9f1f640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 0x7f63d4145150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00c730 tx=0x7f63cc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.941+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc003b70 con 0x7f63d415ceb0 2024-03-29T04:09:19.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.941+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63cc003d10 con 0x7f63d415ceb0 2024-03-29T04:09:19.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.941+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc005310 con 0x7f63d415ceb0 2024-03-29T04:09:19.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.941+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d4143a40 con 0x7f63d415ceb0 2024-03-29T04:09:19.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.941+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d4144000 con 0x7f63d415ceb0 2024-03-29T04:09:19.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.942+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d41444e0 con 0x7f63d415ceb0 2024-03-29T04:09:19.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.942+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f63cc0056c0 con 0x7f63d415ceb0 2024-03-29T04:09:19.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.943+0000 7f63c9ffb640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f63b806c530 0x7f63b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:19.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.943+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f63cc08b720 con 0x7f63d415ceb0 2024-03-29T04:09:19.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.943+0000 7f63d971e640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f63b806c530 0x7f63b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:19.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.943+0000 7f63d971e640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f63b806c530 0x7f63b806e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63d0006fd0 tx=0x7f63d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:19.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:19.947+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f63d41444e0 con 0x7f63d415ceb0 2024-03-29T04:09:20.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.127+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f63d41456f0 con 0x7f63d415ceb0 2024-03-29T04:09:20.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.127+0000 7f63c9ffb640 1 -- 172.21.15.156:0/3135218680 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f63cc059d40 con 0x7f63d415ceb0 2024-03-29T04:09:20.108 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f63b806c530 msgr2=0x7f63b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f63b806c530 0x7f63b806e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63d0006fd0 tx=0x7f63d0008040 comp rx=0 tx=0).stop 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 msgr2=0x7f63d4145150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 0x7f63d4145150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00c730 tx=0x7f63cc00cbf0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 shutdown_connections 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f63b806c530 0x7f63b806e9f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63d415ceb0 0x7f63d4145150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63d4141ca0 0x7f63d4144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 --2- 172.21.15.156:0/3135218680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63d4071020 0x7f63d4140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 >> 172.21.15.156:0/3135218680 conn(0x7f63d40b3eb0 msgr2=0x7f63d415fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:20.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 shutdown_connections 2024-03-29T04:09:20.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.129+0000 7f63db982640 1 -- 172.21.15.156:0/3135218680 wait complete. 2024-03-29T04:09:20.124 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:20.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.245+0000 7f42c3e0d640 1 Processor -- start 2024-03-29T04:09:20.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.245+0000 7f42c3e0d640 1 -- start start 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c3e0d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f42bc15e730 0x7f42bc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c3e0d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f42bc0cf3e0 0x7f42bc0d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c3e0d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42bc0cfa20 con 0x7f42bc0d4800 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c3e0d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f42bc06c720 con 0x7f42bc0cf3e0 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c3e0d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f42bc0cfd00 con 0x7f42bc15e730 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c13a8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f42bc0cf3e0 0x7f42bc0d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c13a8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f42bc0cf3e0 0x7f42bc0d42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43848/0 (socket says 172.21.15.156:43848) 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.246+0000 7f42c13a8640 1 -- 172.21.15.156:0/3420901447 learned_addr learned my addr 172.21.15.156:0/3420901447 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c1ba9640 1 --2- 172.21.15.156:0/3420901447 >> v2:172.21.15.203:3300/0 conn(0x7f42bc15e730 0x7f42bc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c13a8640 1 -- 172.21.15.156:0/3420901447 >> v2:172.21.15.203:3300/0 conn(0x7f42bc15e730 msgr2=0x7f42bc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c13a8640 1 --2- 172.21.15.156:0/3420901447 >> v2:172.21.15.203:3300/0 conn(0x7f42bc15e730 0x7f42bc0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c13a8640 1 -- 172.21.15.156:0/3420901447 >> v1:172.21.15.156:6789/0 conn(0x7f42bc0d4800 legacy=0x7f42bc0d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c13a8640 1 -- 172.21.15.156:0/3420901447 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc0cffe0 con 0x7f42bc0cf3e0 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c0ba7640 1 -- 172.21.15.156:0/3420901447 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822682094 0 0) 0x7f42bc0cfa20 con 0x7f42bc0d4800 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c1ba9640 1 --2- 172.21.15.156:0/3420901447 >> v2:172.21.15.203:3300/0 conn(0x7f42bc15e730 0x7f42bc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.247+0000 7f42c13a8640 1 --2- 172.21.15.156:0/3420901447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc0cf3e0 0x7f42bc0d42c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0d1f20 tx=0x7f42ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51925b7b791ffe78 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c0ba7640 1 -- 172.21.15.156:0/3420901447 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac004230 con 0x7f42bc0cf3e0 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c0ba7640 1 -- 172.21.15.156:0/3420901447 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ac0043d0 con 0x7f42bc0cf3e0 2024-03-29T04:09:20.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c0ba7640 1 -- 172.21.15.156:0/3420901447 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac030840 con 0x7f42bc0cf3e0 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 -- 172.21.15.156:0/3420901447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc0cf3e0 msgr2=0x7f42bc0d42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/3420901447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc0cf3e0 0x7f42bc0d42c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0d1f20 tx=0x7f42ac02b980 comp rx=0 tx=0).stop 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 -- 172.21.15.156:0/3420901447 shutdown_connections 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/3420901447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc0cf3e0 0x7f42bc0d42c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/3420901447 >> v2:172.21.15.203:3300/0 conn(0x7f42bc15e730 0x7f42bc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 -- 172.21.15.156:0/3420901447 >> 172.21.15.156:0/3420901447 conn(0x7f42bc08bfd0 msgr2=0x7f42bc09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 -- 172.21.15.156:0/3420901447 shutdown_connections 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.248+0000 7f42c3e0d640 1 -- 172.21.15.156:0/3420901447 wait complete. 2024-03-29T04:09:20.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.250+0000 7f42c3e0d640 1 Processor -- start 2024-03-29T04:09:20.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.250+0000 7f42c3e0d640 1 -- start start 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c3e0d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f42bc0cf3e0 0x7f42bc1f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c3e0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42bc0d4800 0x7f42bc1f7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c3e0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 0x7f42bc1f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c3e0d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0d2220 con 0x7f42bc15e730 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c3e0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0d4c30 con 0x7f42bc0d4800 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c3e0d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc15eb10 con 0x7f42bc0cf3e0 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c23aa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 0x7f42bc1f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c23aa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 0x7f42bc1f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43858/0 (socket says 172.21.15.156:43858) 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.251+0000 7f42c23aa640 1 -- 172.21.15.156:0/28019959 learned_addr learned my addr 172.21.15.156:0/28019959 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:20.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c13a8640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42bc0d4800 0x7f42bc1f7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c1ba9640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f42bc0cf3e0 0x7f42bc1f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c23aa640 1 -- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f42bc0cf3e0 msgr2=0x7f42bc1f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c23aa640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f42bc0cf3e0 0x7f42bc1f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c23aa640 1 -- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42bc0d4800 msgr2=0x7f42bc1f7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c23aa640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42bc0d4800 0x7f42bc1f7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c23aa640 1 -- 172.21.15.156:0/28019959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc1f82b0 con 0x7f42bc15e730 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c13a8640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42bc0d4800 0x7f42bc1f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c23aa640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 0x7f42bc1f4bd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f42b400c790 tx=0x7f42b400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4003da0 con 0x7f42bc15e730 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b4005070 con 0x7f42bc15e730 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4005460 con 0x7f42bc15e730 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc1f8590 con 0x7f42bc15e730 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.252+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc0c90d0 con 0x7f42bc15e730 2024-03-29T04:09:20.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.253+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42bc0d42c0 con 0x7f42bc15e730 2024-03-29T04:09:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.254+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f42b4005600 con 0x7f42bc15e730 2024-03-29T04:09:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.254+0000 7f42b27fc640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f42a006c560 0x7f42a006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.254+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f42b408b530 con 0x7f42bc15e730 2024-03-29T04:09:20.236 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.254+0000 7f42c1ba9640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f42a006c560 0x7f42a006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.255+0000 7f42c1ba9640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f42a006c560 0x7f42a006ea20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f42b8006fd0 tx=0x7f42b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.259+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f42bc0d42c0 con 0x7f42bc15e730 2024-03-29T04:09:20.422 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.441+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42bc06d9d0 con 0x7f42bc15e730 2024-03-29T04:09:20.423 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.442+0000 7f42b27fc640 1 -- 172.21.15.156:0/28019959 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f42b4059be0 con 0x7f42bc15e730 2024-03-29T04:09:20.423 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:20.423 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.444+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f42a006c560 msgr2=0x7f42a006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.444+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f42a006c560 0x7f42a006ea20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f42b8006fd0 tx=0x7f42b8008040 comp rx=0 tx=0).stop 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.444+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 msgr2=0x7f42bc1f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.444+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 0x7f42bc1f4bd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f42b400c790 tx=0x7f42b400cc50 comp rx=0 tx=0).stop 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 shutdown_connections 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f42a006c560 0x7f42a006ea20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42bc15e730 0x7f42bc1f4bd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42bc0d4800 0x7f42bc1f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 --2- 172.21.15.156:0/28019959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f42bc0cf3e0 0x7f42bc1f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 >> 172.21.15.156:0/28019959 conn(0x7f42bc08bfd0 msgr2=0x7f42bc06def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 shutdown_connections 2024-03-29T04:09:20.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.445+0000 7f42c3e0d640 1 -- 172.21.15.156:0/28019959 wait complete. 2024-03-29T04:09:20.436 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:20.437 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:20.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.561+0000 7f2a4354d640 1 Processor -- start 2024-03-29T04:09:20.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.561+0000 7f2a4354d640 1 -- start start 2024-03-29T04:09:20.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.562+0000 7f2a4354d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2a3c0b25f0 0x7f2a3c0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a4354d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a3c0b6370 con 0x7f2a3c161e40 2024-03-29T04:09:20.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a4354d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a3c0b4190 con 0x7f2a3c06bcc0 2024-03-29T04:09:20.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a4354d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2a3c0b6120 con 0x7f2a3c0b25f0 2024-03-29T04:09:20.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a41aea640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2a3c0b25f0 0x7f2a3c0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a412e9640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2a3c161e40 0x7f2a3c06b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47162/0 (socket says 172.21.15.156:47162) 2024-03-29T04:09:20.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a412e9640 1 -- 172.21.15.156:0/3081494686 learned_addr learned my addr 172.21.15.156:0/3081494686 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:20.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a41aea640 1 -- 172.21.15.156:0/3081494686 >> v1:172.21.15.203:6789/0 conn(0x7f2a3c06bcc0 legacy=0x7f2a3c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a41aea640 1 -- 172.21.15.156:0/3081494686 >> v1:172.21.15.156:6789/0 conn(0x7f2a3c161e40 legacy=0x7f2a3c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.563+0000 7f2a41aea640 1 -- 172.21.15.156:0/3081494686 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c0b4470 con 0x7f2a3c0b25f0 2024-03-29T04:09:20.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.564+0000 7f2a41aea640 1 --2- 172.21.15.156:0/3081494686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c0b5be0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0097a0 tx=0x7f2a2c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb91a75260c35fe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.564+0000 7f2a3bfff640 1 -- 172.21.15.156:0/3081494686 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c0315a0 con 0x7f2a3c0b25f0 2024-03-29T04:09:20.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.564+0000 7f2a3bfff640 1 -- 172.21.15.156:0/3081494686 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c037070 con 0x7f2a3c0b25f0 2024-03-29T04:09:20.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.564+0000 7f2a3bfff640 1 -- 172.21.15.156:0/3081494686 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c031960 con 0x7f2a3c0b25f0 2024-03-29T04:09:20.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 -- 172.21.15.156:0/3081494686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c0b25f0 msgr2=0x7f2a3c0b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 --2- 172.21.15.156:0/3081494686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c0b5be0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0097a0 tx=0x7f2a2c02b840 comp rx=0 tx=0).stop 2024-03-29T04:09:20.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 -- 172.21.15.156:0/3081494686 shutdown_connections 2024-03-29T04:09:20.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 --2- 172.21.15.156:0/3081494686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c0b5be0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 -- 172.21.15.156:0/3081494686 >> 172.21.15.156:0/3081494686 conn(0x7f2a3c0b6580 msgr2=0x7f2a3c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:20.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 -- 172.21.15.156:0/3081494686 shutdown_connections 2024-03-29T04:09:20.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.565+0000 7f2a4354d640 1 -- 172.21.15.156:0/3081494686 wait complete. 2024-03-29T04:09:20.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.566+0000 7f2a4354d640 1 Processor -- start 2024-03-29T04:09:20.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.566+0000 7f2a4354d640 1 -- start start 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a3c06bcc0 0x7f2a3c1e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c143750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 0x7f2a3c143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0aee10 con 0x7f2a3c161e40 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0b1420 con 0x7f2a3c0b25f0 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3c0aef90 con 0x7f2a3c06bcc0 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 0x7f2a3c143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 0x7f2a3c143c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43884/0 (socket says 172.21.15.156:43884) 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 -- 172.21.15.156:0/1640455081 learned_addr learned my addr 172.21.15.156:0/1640455081 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a40ae8640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a412e9640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a3c06bcc0 0x7f2a3c1e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 -- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a3c06bcc0 msgr2=0x7f2a3c1e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a3c06bcc0 0x7f2a3c1e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 -- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a3c0b25f0 msgr2=0x7f2a3c143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 -- 172.21.15.156:0/1640455081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c1e8850 con 0x7f2a3c161e40 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a41aea640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 0x7f2a3c143c90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c034040 tx=0x7f2a2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c032840 con 0x7f2a3c161e40 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c0329e0 con 0x7f2a3c161e40 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c1e7cd0 con 0x7f2a3c161e40 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.567+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c032dd0 con 0x7f2a3c161e40 2024-03-29T04:09:20.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.568+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c1e8290 con 0x7f2a3c161e40 2024-03-29T04:09:20.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.569+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c09f3c0 con 0x7f2a3c161e40 2024-03-29T04:09:20.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.569+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2a2c03b3f0 con 0x7f2a3c161e40 2024-03-29T04:09:20.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.569+0000 7f2a39ffb640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2a2806c530 0x7f2a2806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.570+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2a2c02f030 con 0x7f2a3c161e40 2024-03-29T04:09:20.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.570+0000 7f2a412e9640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2a2806c530 0x7f2a2806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.570+0000 7f2a412e9640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2a2806c530 0x7f2a2806e9f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2a34005c00 tx=0x7f2a34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.575+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2a2c040070 con 0x7f2a3c161e40 2024-03-29T04:09:20.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.754+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a3c04f9d0 con 0x7f2a3c161e40 2024-03-29T04:09:20.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.755+0000 7f2a39ffb640 1 -- 172.21.15.156:0/1640455081 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f2a2c07bd10 con 0x7f2a3c161e40 2024-03-29T04:09:20.735 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:20.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2a2806c530 msgr2=0x7f2a2806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2a2806c530 0x7f2a2806e9f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2a34005c00 tx=0x7f2a34009040 comp rx=0 tx=0).stop 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 msgr2=0x7f2a3c143c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 0x7f2a3c143c90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c034040 tx=0x7f2a2c0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 shutdown_connections 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2a2806c530 0x7f2a2806e9f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a3c161e40 0x7f2a3c143c90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a3c0b25f0 0x7f2a3c143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 --2- 172.21.15.156:0/1640455081 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a3c06bcc0 0x7f2a3c1e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 >> 172.21.15.156:0/1640455081 conn(0x7f2a3c0b6580 msgr2=0x7f2a3c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 shutdown_connections 2024-03-29T04:09:20.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.757+0000 7f2a4354d640 1 -- 172.21.15.156:0/1640455081 wait complete. 2024-03-29T04:09:20.748 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:20.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.873+0000 7f4618488640 1 Processor -- start 2024-03-29T04:09:20.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.873+0000 7f4618488640 1 -- start start 2024-03-29T04:09:20.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4618488640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46100b44e0 0x7f46100cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4618488640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 0x7f46100d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4618488640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461015cb10 con 0x7f4610145c70 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4618488640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f461015cdf0 con 0x7f46100b44e0 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4618488640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f461015d0d0 con 0x7f46100cf340 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4616224640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4610145c70 0x7f46100b3dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47168/0 (socket says 172.21.15.156:47168) 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4616224640 1 -- 172.21.15.156:0/1872760285 learned_addr learned my addr 172.21.15.156:0/1872760285 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792923886 0 0) 0x7f461015cb10 con 0x7f4610145c70 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4615a23640 1 --2- 172.21.15.156:0/1872760285 >> v2:172.21.15.203:3300/0 conn(0x7f46100b44e0 0x7f46100cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45fc003520 con 0x7f4610145c70 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4616a25640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 0x7f46100d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.875+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 833686257 0 0) 0x7f45fc003520 con 0x7f4610145c70 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 msgr2=0x7f46100d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 0x7f46100d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 >> v2:172.21.15.203:3300/0 conn(0x7f46100b44e0 msgr2=0x7f46100cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:20.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 --2- 172.21.15.156:0/1872760285 >> v2:172.21.15.203:3300/0 conn(0x7f46100b44e0 0x7f46100cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461015d3b0 con 0x7f4610145c70 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f460c001670 con 0x7f4610145c70 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 >> v1:172.21.15.156:6789/0 conn(0x7f4610145c70 legacy=0x7f46100b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc0039b0 0x7f45fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 0x7f45fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc004d40 0x7f45fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f461015cdf0 con 0x7f45fc004d40 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f461015d0d0 con 0x7f45fc0039b0 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f45fc00b9c0 con 0x7f46100cf340 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616a25640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc004d40 0x7f45fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4615a23640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 0x7f45fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616224640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc0039b0 0x7f45fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616a25640 1 -- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 msgr2=0x7f45fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616a25640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46100cf340 0x7f45fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616a25640 1 -- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc0039b0 msgr2=0x7f45fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:20.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616a25640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc0039b0 0x7f45fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.876+0000 7f4616a25640 1 -- 172.21.15.156:0/1872760285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45fc003520 con 0x7f45fc004d40 2024-03-29T04:09:20.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.877+0000 7f4616a25640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc004d40 0x7f45fc0051c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4608009fc0 tx=0x7f45fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672cd8ddc5f1ba9a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.877+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608039070 con 0x7f45fc004d40 2024-03-29T04:09:20.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.877+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460802cc20 con 0x7f45fc004d40 2024-03-29T04:09:20.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.877+0000 7f4615222640 1 -- 172.21.15.156:0/1872760285 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608009c10 con 0x7f45fc004d40 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4618488640 1 -- 172.21.15.156:0/1872760285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc004d40 msgr2=0x7f45fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4618488640 1 --2- 172.21.15.156:0/1872760285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc004d40 0x7f45fc0051c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4608009fc0 tx=0x7f45fc005830 comp rx=0 tx=0).stop 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4616a25640 1 -- 172.21.15.156:0/1872760285 reap_dead start 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4618488640 1 -- 172.21.15.156:0/1872760285 shutdown_connections 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4618488640 1 -- 172.21.15.156:0/1872760285 >> 172.21.15.156:0/1872760285 conn(0x7f46100b7660 msgr2=0x7f46100b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4618488640 1 -- 172.21.15.156:0/1872760285 shutdown_connections 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.878+0000 7f4618488640 1 -- 172.21.15.156:0/1872760285 wait complete. 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.880+0000 7f4618488640 1 Processor -- start 2024-03-29T04:09:20.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.880+0000 7f4618488640 1 -- start start 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4618488640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45fc004d40 0x7f46101544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4618488640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46100ce240 0x7f4610144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4618488640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 0x7f4610155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4618488640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46101461d0 con 0x7f4610157d80 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4618488640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4610146050 con 0x7f46100ce240 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4618488640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f46100d0ef0 con 0x7f45fc004d40 2024-03-29T04:09:20.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 0x7f4610155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 0x7f4610155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43900/0 (socket says 172.21.15.156:43900) 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 -- 172.21.15.156:0/2068054121 learned_addr learned my addr 172.21.15.156:0/2068054121 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616224640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45fc004d40 0x7f46101544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4615a23640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46100ce240 0x7f4610144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 -- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45fc004d40 msgr2=0x7f46101544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45fc004d40 0x7f46101544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 -- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46100ce240 msgr2=0x7f4610144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46100ce240 0x7f4610144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 -- 172.21.15.156:0/2068054121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4610154a00 con 0x7f4610157d80 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616224640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45fc004d40 0x7f46101544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4615a23640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46100ce240 0x7f4610144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.881+0000 7f4616a25640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 0x7f4610155b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f460800e230 tx=0x7f4608009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.882+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608039070 con 0x7f4610157d80 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.882+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4608044070 con 0x7f4610157d80 2024-03-29T04:09:20.865 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.882+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4610154ca0 con 0x7f4610157d80 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.882+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460803e810 con 0x7f4610157d80 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.882+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46101e6080 con 0x7f4610157d80 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.883+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4610070760 con 0x7f4610157d80 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.883+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4608005d10 con 0x7f4610157d80 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.883+0000 7f4606ffd640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f45f406c530 0x7f45f406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.884+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f46080ad880 con 0x7f4610157d80 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.884+0000 7f4616224640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f45f406c530 0x7f45f406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:20.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.884+0000 7f4616224640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f45f406c530 0x7f45f406e9f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f460c001550 tx=0x7f460c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:20.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:20.888+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4610070760 con 0x7f4610157d80 2024-03-29T04:09:21.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.069+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4610070e10 con 0x7f4610157d80 2024-03-29T04:09:21.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.070+0000 7f4606ffd640 1 -- 172.21.15.156:0/2068054121 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f460804d020 con 0x7f4610157d80 2024-03-29T04:09:21.050 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:21.052 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.071+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f45f406c530 msgr2=0x7f45f406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.052 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.071+0000 7f4618488640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f45f406c530 0x7f45f406e9f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f460c001550 tx=0x7f460c009040 comp rx=0 tx=0).stop 2024-03-29T04:09:21.052 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.071+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 msgr2=0x7f4610155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.071+0000 7f4618488640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 0x7f4610155b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f460800e230 tx=0x7f4608009750 comp rx=0 tx=0).stop 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 shutdown_connections 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f45f406c530 0x7f45f406e9f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4610157d80 0x7f4610155b60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46100ce240 0x7f4610144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 --2- 172.21.15.156:0/2068054121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f45fc004d40 0x7f46101544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 >> 172.21.15.156:0/2068054121 conn(0x7f46100b7660 msgr2=0x7f46100b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 shutdown_connections 2024-03-29T04:09:21.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.072+0000 7f4618488640 1 -- 172.21.15.156:0/2068054121 wait complete. 2024-03-29T04:09:21.064 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:21.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.188+0000 7f5d04055640 1 Processor -- start 2024-03-29T04:09:21.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.188+0000 7f5d04055640 1 -- start start 2024-03-29T04:09:21.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d04055640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5cfc0d1550 0x7f5cfc169300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d04055640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc06c500 con 0x7f5cfc0d1ba0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d04055640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc06c7e0 con 0x7f5cfc161be0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d04055640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5cfc0d3140 con 0x7f5cfc0d1550 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d015f0640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5cfc0d1ba0 0x7f5cfc0d0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47178/0 (socket says 172.21.15.156:47178) 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d015f0640 1 -- 172.21.15.156:0/1475498695 learned_addr learned my addr 172.21.15.156:0/1475498695 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d025f2640 1 --2- 172.21.15.156:0/1475498695 >> v2:172.21.15.203:3300/0 conn(0x7f5cfc0d1550 0x7f5cfc169300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147893603 0 0) 0x7f5cfc06c500 con 0x7f5cfc0d1ba0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.189+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce8003520 con 0x7f5cfc0d1ba0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 687656551 0 0) 0x7f5ce8003520 con 0x7f5cfc0d1ba0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 >> v2:172.21.15.203:3300/0 conn(0x7f5cfc0d1550 msgr2=0x7f5cfc169300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 --2- 172.21.15.156:0/1475498695 >> v2:172.21.15.203:3300/0 conn(0x7f5cfc0d1550 0x7f5cfc169300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 >> v1:172.21.15.203:6789/0 conn(0x7f5cfc161be0 legacy=0x7f5cfc0d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc06cac0 con 0x7f5cfc0d1ba0 2024-03-29T04:09:21.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f5cf8001670 con 0x7f5cfc0d1ba0 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 >> v1:172.21.15.156:6789/0 conn(0x7f5cfc0d1ba0 legacy=0x7f5cfc0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80051c0 0x7f5ce800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc0d3140 con 0x7f5ce8004800 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce800c160 con 0x7f5ce80051c0 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce800c360 con 0x7f5ce80039b0 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d015f0640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d025f2640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80051c0 0x7f5ce800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d01df1640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d015f0640 1 -- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 msgr2=0x7f5ce8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d015f0640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d015f0640 1 -- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80051c0 msgr2=0x7f5ce800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d015f0640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce80051c0 0x7f5ce800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d015f0640 1 -- 172.21.15.156:0/1475498695 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce8003520 con 0x7f5ce8004800 2024-03-29T04:09:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d025f2640 1 -- 172.21.15.156:0/1475498695 reap_dead start 2024-03-29T04:09:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.190+0000 7f5d01df1640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce80039b0 0x7f5ce8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.191+0000 7f5d015f0640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c210 tx=0x7f5ce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5043aa49d3e7ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.191+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8007cf0 con 0x7f5ce8004800 2024-03-29T04:09:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.191+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8030070 con 0x7f5ce8004800 2024-03-29T04:09:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.191+0000 7f5d00def640 1 -- 172.21.15.156:0/1475498695 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8005d50 con 0x7f5ce8004800 2024-03-29T04:09:21.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 -- 172.21.15.156:0/1475498695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 msgr2=0x7f5ce8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c210 tx=0x7f5ce8005830 comp rx=0 tx=0).stop 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 -- 172.21.15.156:0/1475498695 shutdown_connections 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 --2- 172.21.15.156:0/1475498695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ce8004800 0x7f5ce8004c80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 -- 172.21.15.156:0/1475498695 >> 172.21.15.156:0/1475498695 conn(0x7f5cfc0d4450 msgr2=0x7f5cfc0d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 -- 172.21.15.156:0/1475498695 shutdown_connections 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.192+0000 7f5d04055640 1 -- 172.21.15.156:0/1475498695 wait complete. 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.193+0000 7f5d04055640 1 Processor -- start 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.193+0000 7f5d04055640 1 -- start start 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5cfc0b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 0x7f5cfc157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cfc155f70 0x7f5cfc1563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc0d3360 con 0x7f5cfc15ac00 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc06ce30 con 0x7f5cfc155f70 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5cfc0d0e40 con 0x7f5ce8004800 2024-03-29T04:09:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 0x7f5cfc157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 0x7f5cfc157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43922/0 (socket says 172.21.15.156:43922) 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 -- 172.21.15.156:0/3220680509 learned_addr learned my addr 172.21.15.156:0/3220680509 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d025f2640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cfc155f70 0x7f5cfc1563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d01df1640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5cfc0b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 -- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 msgr2=0x7f5cfc0b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5cfc0b8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 -- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cfc155f70 msgr2=0x7f5cfc1563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cfc155f70 0x7f5cfc1563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 -- 172.21.15.156:0/3220680509 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc0b8990 con 0x7f5cfc15ac00 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d015f0640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 0x7f5cfc157e40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c330 tx=0x7f5cfc06c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8046070 con 0x7f5cfc15ac00 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8004bf0 con 0x7f5cfc15ac00 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8001c30 con 0x7f5cfc15ac00 2024-03-29T04:09:21.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.194+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cfc0b8c70 con 0x7f5cfc15ac00 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.195+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cfc0b9230 con 0x7f5cfc15ac00 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.196+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cfc0a23a0 con 0x7f5cfc15ac00 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.196+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5cf8007070 con 0x7f5cfc15ac00 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.197+0000 7f5cee7fc640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5ce006c560 0x7f5ce006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.197+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5cf80aca90 con 0x7f5cfc15ac00 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.197+0000 7f5d01df1640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5ce006c560 0x7f5ce006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.197+0000 7f5d01df1640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5ce006c560 0x7f5ce006ea20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0511f0 tx=0x7f5cf0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.202+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5cf807a9d0 con 0x7f5cfc15ac00 2024-03-29T04:09:21.360 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.380+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5cfc0a23a0 con 0x7f5cfc15ac00 2024-03-29T04:09:21.360 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.380+0000 7f5cee7fc640 1 -- 172.21.15.156:0/3220680509 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f5cf807ecd0 con 0x7f5cfc15ac00 2024-03-29T04:09:21.361 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5ce006c560 msgr2=0x7f5ce006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5ce006c560 0x7f5ce006ea20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0511f0 tx=0x7f5cf0009040 comp rx=0 tx=0).stop 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 msgr2=0x7f5cfc157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 0x7f5cfc157e40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c330 tx=0x7f5cfc06c9b0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 shutdown_connections 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5ce006c560 0x7f5ce006ea20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cfc155f70 0x7f5cfc1563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cfc15ac00 0x7f5cfc157e40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 --2- 172.21.15.156:0/3220680509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ce8004800 0x7f5cfc0b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 >> 172.21.15.156:0/3220680509 conn(0x7f5cfc0d4450 msgr2=0x7f5cfc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 shutdown_connections 2024-03-29T04:09:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.383+0000 7f5d04055640 1 -- 172.21.15.156:0/3220680509 wait complete. 2024-03-29T04:09:21.374 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:21.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.494+0000 7fbce3f96640 1 Processor -- start 2024-03-29T04:09:21.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.494+0000 7fbce3f96640 1 -- start start 2024-03-29T04:09:21.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce3f96640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbcdc15e730 0x7fbcdc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce3f96640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc0cf3e0 0x7fbcdc0d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce3f96640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcdc0d4c90 con 0x7fbcdc0d4800 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce3f96640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbcdc06c720 con 0x7fbcdc15e730 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce3f96640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbcdc0cfa20 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce1d32640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbcdc15e730 0x7fbcdc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce2533640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbcdc0d4800 0x7fbcdc0d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47192/0 (socket says 172.21.15.156:47192) 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce2533640 1 -- 172.21.15.156:0/3143096914 learned_addr learned my addr 172.21.15.156:0/3143096914 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce1531640 1 --2- 172.21.15.156:0/3143096914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc0cf3e0 0x7fbcdc0d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce1d32640 1 -- 172.21.15.156:0/3143096914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc0cf3e0 msgr2=0x7fbcdc0d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce1d32640 1 --2- 172.21.15.156:0/3143096914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc0cf3e0 0x7fbcdc0d42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce1d32640 1 -- 172.21.15.156:0/3143096914 >> v1:172.21.15.156:6789/0 conn(0x7fbcdc0d4800 legacy=0x7fbcdc0d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce0d30640 1 -- 172.21.15.156:0/3143096914 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011656485 0 0) 0x7fbcdc0d4c90 con 0x7fbcdc0d4800 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.495+0000 7fbce1d32640 1 -- 172.21.15.156:0/3143096914 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc0cfd00 con 0x7fbcdc15e730 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.496+0000 7fbce1d32640 1 --2- 172.21.15.156:0/3143096914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc15e730 0x7fbcdc0ceea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80097a0 tx=0x7fbcd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc056f4cc189b360 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.496+0000 7fbce0d30640 1 -- 172.21.15.156:0/3143096914 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8035020 con 0x7fbcdc15e730 2024-03-29T04:09:21.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.496+0000 7fbce0d30640 1 -- 172.21.15.156:0/3143096914 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd8031830 con 0x7fbcdc15e730 2024-03-29T04:09:21.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.496+0000 7fbce0d30640 1 -- 172.21.15.156:0/3143096914 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8031c00 con 0x7fbcdc15e730 2024-03-29T04:09:21.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 -- 172.21.15.156:0/3143096914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc15e730 msgr2=0x7fbcdc0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 --2- 172.21.15.156:0/3143096914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc15e730 0x7fbcdc0ceea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80097a0 tx=0x7fbcd802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 -- 172.21.15.156:0/3143096914 shutdown_connections 2024-03-29T04:09:21.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 --2- 172.21.15.156:0/3143096914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc0cf3e0 0x7fbcdc0d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 --2- 172.21.15.156:0/3143096914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc15e730 0x7fbcdc0ceea0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 -- 172.21.15.156:0/3143096914 >> 172.21.15.156:0/3143096914 conn(0x7fbcdc08bfd0 msgr2=0x7fbcdc09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 -- 172.21.15.156:0/3143096914 shutdown_connections 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.497+0000 7fbce3f96640 1 -- 172.21.15.156:0/3143096914 wait complete. 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.498+0000 7fbce3f96640 1 Processor -- start 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.498+0000 7fbce3f96640 1 -- start start 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 0x7fbcdc153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcdc0d4800 0x7fbcdc0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc15e730 0x7fbcdc0b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc0d2220 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc0d02c0 con 0x7fbcdc0d4800 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbcdc15eb10 con 0x7fbcdc15e730 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 0x7fbcdc153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 0x7fbcdc153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43930/0 (socket says 172.21.15.156:43930) 2024-03-29T04:09:21.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 -- 172.21.15.156:0/699909539 learned_addr learned my addr 172.21.15.156:0/699909539 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1531640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcdc0d4800 0x7fbcdc0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce2533640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc15e730 0x7fbcdc0b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 -- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc15e730 msgr2=0x7fbcdc0b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc15e730 0x7fbcdc0b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 -- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcdc0d4800 msgr2=0x7fbcdc0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcdc0d4800 0x7fbcdc0ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 -- 172.21.15.156:0/699909539 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc0ca8c0 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce2533640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc15e730 0x7fbcdc0b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce1d32640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 0x7fbcdc153cf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8034040 tx=0x7fbcd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8040070 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd8039a20 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8035040 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcdc15fab0 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.499+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcdc160070 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.501+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcdc068610 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.501+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbcd8038020 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.501+0000 7fbcd27fc640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbcc006c530 0x7fbcc006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.501+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbcd80ad0d0 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.501+0000 7fbce1531640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbcc006c530 0x7fbcc006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.502+0000 7fbce1531640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbcc006c530 0x7fbcc006e9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc06ccd0 tx=0x7fbccc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.505+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbcd807b780 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.689+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbcdc0b2ae0 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.690+0000 7fbcd27fc640 1 -- 172.21.15.156:0/699909539 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fbcd807b080 con 0x7fbcdc0cf3e0 2024-03-29T04:09:21.671 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:21.671 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbcc006c530 msgr2=0x7fbcc006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbcc006c530 0x7fbcc006e9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc06ccd0 tx=0x7fbccc008040 comp rx=0 tx=0).stop 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 msgr2=0x7fbcdc153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 0x7fbcdc153cf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8034040 tx=0x7fbcd80046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 shutdown_connections 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbcc006c530 0x7fbcc006e9f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbcdc15e730 0x7fbcdc0b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcdc0d4800 0x7fbcdc0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 --2- 172.21.15.156:0/699909539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcdc0cf3e0 0x7fbcdc153cf0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 >> 172.21.15.156:0/699909539 conn(0x7fbcdc08bfd0 msgr2=0x7fbcdc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 shutdown_connections 2024-03-29T04:09:21.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.693+0000 7fbce3f96640 1 -- 172.21.15.156:0/699909539 wait complete. 2024-03-29T04:09:21.684 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:21.684 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:21.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 Processor -- start 2024-03-29T04:09:21.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 -- start start 2024-03-29T04:09:21.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac15e920 0x7f9dac0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9dac0cfc30 0x7f9dac0d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac0cf190 con 0x7f9dac0d2120 2024-03-29T04:09:21.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9dac0cf7a0 con 0x7f9dac0cfc30 2024-03-29T04:09:21.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.822+0000 7f9db3b15640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9dac0d3080 con 0x7f9dac15e920 2024-03-29T04:09:21.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db20b2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9dac0cfc30 0x7f9dac0d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db18b1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac15e920 0x7f9dac0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db10b0640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9dac0d2120 0x7f9dac06c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:55358/0 (socket says 172.21.15.156:55358) 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db10b0640 1 -- 172.21.15.156:0/4019786756 learned_addr learned my addr 172.21.15.156:0/4019786756 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db18b1640 1 -- 172.21.15.156:0/4019786756 >> v2:172.21.15.203:3300/0 conn(0x7f9dac0cfc30 msgr2=0x7f9dac0d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db18b1640 1 --2- 172.21.15.156:0/4019786756 >> v2:172.21.15.203:3300/0 conn(0x7f9dac0cfc30 0x7f9dac0d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db18b1640 1 -- 172.21.15.156:0/4019786756 >> v1:172.21.15.203:6789/0 conn(0x7f9dac0d2120 legacy=0x7f9dac06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db18b1640 1 -- 172.21.15.156:0/4019786756 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac0cf430 con 0x7f9dac15e920 2024-03-29T04:09:21.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.823+0000 7f9db20b2640 1 --2- 172.21.15.156:0/4019786756 >> v2:172.21.15.203:3300/0 conn(0x7f9dac0cfc30 0x7f9dac0d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db18b1640 1 --2- 172.21.15.156:0/4019786756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac15e920 0x7f9dac0d26e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9da80097a0 tx=0x7f9da802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59093d417707ea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db08af640 1 -- 172.21.15.156:0/4019786756 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8035020 con 0x7f9dac15e920 2024-03-29T04:09:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db08af640 1 -- 172.21.15.156:0/4019786756 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9da8031830 con 0x7f9dac15e920 2024-03-29T04:09:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db08af640 1 -- 172.21.15.156:0/4019786756 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8031c00 con 0x7f9dac15e920 2024-03-29T04:09:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 -- 172.21.15.156:0/4019786756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac15e920 msgr2=0x7f9dac0d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 --2- 172.21.15.156:0/4019786756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac15e920 0x7f9dac0d26e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9da80097a0 tx=0x7f9da802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 -- 172.21.15.156:0/4019786756 shutdown_connections 2024-03-29T04:09:21.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 --2- 172.21.15.156:0/4019786756 >> v2:172.21.15.203:3300/0 conn(0x7f9dac0cfc30 0x7f9dac0d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 --2- 172.21.15.156:0/4019786756 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac15e920 0x7f9dac0d26e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 -- 172.21.15.156:0/4019786756 >> 172.21.15.156:0/4019786756 conn(0x7f9dac09d7a0 msgr2=0x7f9dac0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 -- 172.21.15.156:0/4019786756 shutdown_connections 2024-03-29T04:09:21.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.825+0000 7f9db3b15640 1 -- 172.21.15.156:0/4019786756 wait complete. 2024-03-29T04:09:21.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 Processor -- start 2024-03-29T04:09:21.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 -- start start 2024-03-29T04:09:21.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac0cfc30 0x7f9dac1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dac0d2120 0x7f9dac15ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 0x7f9dac15f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac0d34e0 con 0x7f9dac15e920 2024-03-29T04:09:21.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac0d3360 con 0x7f9dac0d2120 2024-03-29T04:09:21.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db3b15640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9dac0d2c20 con 0x7f9dac0cfc30 2024-03-29T04:09:21.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db20b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 0x7f9dac15f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db20b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 0x7f9dac15f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42594/0 (socket says 172.21.15.156:42594) 2024-03-29T04:09:21.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db20b2640 1 -- 172.21.15.156:0/1933349737 learned_addr learned my addr 172.21.15.156:0/1933349737 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:21.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db18b1640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac0cfc30 0x7f9dac1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.827+0000 7f9db10b0640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dac0d2120 0x7f9dac15ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db20b2640 1 -- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac0cfc30 msgr2=0x7f9dac1591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db20b2640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac0cfc30 0x7f9dac1591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db20b2640 1 -- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dac0d2120 msgr2=0x7f9dac15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db20b2640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dac0d2120 0x7f9dac15ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db20b2640 1 -- 172.21.15.156:0/1933349737 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac15f8d0 con 0x7f9dac15e920 2024-03-29T04:09:21.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db20b2640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 0x7f9dac15f390 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009f00 tx=0x7f9da400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da4018030 con 0x7f9dac15e920 2024-03-29T04:09:21.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dac15fb80 con 0x7f9dac15e920 2024-03-29T04:09:21.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9da400ce50 con 0x7f9dac15e920 2024-03-29T04:09:21.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da4003bb0 con 0x7f9dac15e920 2024-03-29T04:09:21.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.828+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dac160140 con 0x7f9dac15e920 2024-03-29T04:09:21.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.829+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dac0a2480 con 0x7f9dac15e920 2024-03-29T04:09:21.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.830+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9da4005d10 con 0x7f9dac15e920 2024-03-29T04:09:21.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.830+0000 7f9da27fc640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9d9006c530 0x7f9d9006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:21.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.830+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9da408c070 con 0x7f9dac15e920 2024-03-29T04:09:21.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.830+0000 7f9db18b1640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9d9006c530 0x7f9d9006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:21.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.831+0000 7f9db18b1640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9d9006c530 0x7f9d9006e9f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9da8036000 tx=0x7f9da803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:21.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:21.835+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9da4022070 con 0x7f9dac15e920 2024-03-29T04:09:21.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.014+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9dac15d540 con 0x7f9dac15e920 2024-03-29T04:09:21.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.014+0000 7f9da27fc640 1 -- 172.21.15.156:0/1933349737 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f9da4010040 con 0x7f9dac15e920 2024-03-29T04:09:21.994 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9d9006c530 msgr2=0x7f9d9006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9d9006c530 0x7f9d9006e9f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9da8036000 tx=0x7f9da803c040 comp rx=0 tx=0).stop 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 msgr2=0x7f9dac15f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 0x7f9dac15f390 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009f00 tx=0x7f9da400c490 comp rx=0 tx=0).stop 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 shutdown_connections 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9d9006c530 0x7f9d9006e9f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac15e920 0x7f9dac15f390 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dac0d2120 0x7f9dac15ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 --2- 172.21.15.156:0/1933349737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9dac0cfc30 0x7f9dac1591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 >> 172.21.15.156:0/1933349737 conn(0x7f9dac09d7a0 msgr2=0x7f9dac0b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:21.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 shutdown_connections 2024-03-29T04:09:21.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.016+0000 7f9db3b15640 1 -- 172.21.15.156:0/1933349737 wait complete. 2024-03-29T04:09:22.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:22.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.131+0000 7f9b7ff24640 1 Processor -- start 2024-03-29T04:09:22.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.131+0000 7f9b7ff24640 1 -- start start 2024-03-29T04:09:22.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ff24640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9b780d2120 0x7f9b7806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ff24640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b780d2df0 con 0x7f9b7806cb10 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ff24640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b780cf190 con 0x7f9b7815e920 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ff24640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9b780cf7a0 con 0x7f9b780d2120 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7e4c1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9b7806cb10 0x7f9b780cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57946/0 (socket says 172.21.15.156:57946) 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7e4c1640 1 -- 172.21.15.156:0/4293493708 learned_addr learned my addr 172.21.15.156:0/4293493708 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7d4bf640 1 --2- 172.21.15.156:0/4293493708 >> v2:172.21.15.203:3300/0 conn(0x7f9b780d2120 0x7f9b7806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458828034 0 0) 0x7f9b780d2df0 con 0x7f9b7806cb10 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b64003520 con 0x7f9b7806cb10 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 155762152 0 0) 0x7f9b64003520 con 0x7f9b7806cb10 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 >> v2:172.21.15.203:3300/0 conn(0x7f9b780d2120 msgr2=0x7f9b7806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 --2- 172.21.15.156:0/4293493708 >> v2:172.21.15.203:3300/0 conn(0x7f9b780d2120 0x7f9b7806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 >> v1:172.21.15.203:6789/0 conn(0x7f9b7815e920 legacy=0x7f9b780d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b780d3080 con 0x7f9b7806cb10 2024-03-29T04:09:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f9b700019f0 con 0x7f9b7806cb10 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 >> v1:172.21.15.156:6789/0 conn(0x7f9b7806cb10 legacy=0x7f9b780cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640051c0 0x7f9b6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b780cf7a0 con 0x7f9b640039b0 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6400c060 con 0x7f9b64004800 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.132+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6400c260 con 0x7f9b640051c0 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7d4bf640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7e4c1640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640051c0 0x7f9b6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7dcc0640 1 -- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640051c0 msgr2=0x7f9b6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640051c0 0x7f9b6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7dcc0640 1 -- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b64004800 msgr2=0x7f9b64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7dcc0640 1 -- 172.21.15.156:0/4293493708 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b64003520 con 0x7f9b640039b0 2024-03-29T04:09:22.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.133+0000 7f9b7e4c1640 1 -- 172.21.15.156:0/4293493708 reap_dead start 2024-03-29T04:09:22.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.134+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9b74001750 tx=0x7f9b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd729356e487f242 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.134+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74001370 con 0x7f9b640039b0 2024-03-29T04:09:22.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.134+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b740052d0 con 0x7f9b640039b0 2024-03-29T04:09:22.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.134+0000 7f9b7ccbe640 1 -- 172.21.15.156:0/4293493708 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74003830 con 0x7f9b640039b0 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 -- 172.21.15.156:0/4293493708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b640039b0 msgr2=0x7f9b64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9b74001750 tx=0x7f9b64005830 comp rx=0 tx=0).stop 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 -- 172.21.15.156:0/4293493708 shutdown_connections 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/4293493708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 -- 172.21.15.156:0/4293493708 >> 172.21.15.156:0/4293493708 conn(0x7f9b7809d7a0 msgr2=0x7f9b780d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 -- 172.21.15.156:0/4293493708 shutdown_connections 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.135+0000 7f9b7ff24640 1 -- 172.21.15.156:0/4293493708 wait complete. 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.137+0000 7f9b7ff24640 1 Processor -- start 2024-03-29T04:09:22.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.137+0000 7f9b7ff24640 1 -- start start 2024-03-29T04:09:22.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7ff24640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640039b0 0x7f9b780ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7ff24640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 0x7f9b78158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7ff24640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b781595f0 0x7f9b78160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7ff24640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b780d34e0 con 0x7f9b780cdc10 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7ff24640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b780d3360 con 0x7f9b781595f0 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 0x7f9b78158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7ff24640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7806c5d0 con 0x7f9b640039b0 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 0x7f9b78158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42612/0 (socket says 172.21.15.156:42612) 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 -- 172.21.15.156:0/3287465331 learned_addr learned my addr 172.21.15.156:0/3287465331 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640039b0 0x7f9b780ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7e4c1640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b781595f0 0x7f9b78160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 -- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640039b0 msgr2=0x7f9b780ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640039b0 0x7f9b780ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 -- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b781595f0 msgr2=0x7f9b78160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b781595f0 0x7f9b78160c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.138+0000 7f9b7d4bf640 1 -- 172.21.15.156:0/3287465331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b780cbf00 con 0x7f9b780cdc10 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.139+0000 7f9b7d4bf640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 0x7f9b78158e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9b68011aa0 tx=0x7f9b68011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.139+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6800fb20 con 0x7f9b780cdc10 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.139+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6800fcc0 con 0x7f9b780cdc10 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.139+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b780ce190 con 0x7f9b780cdc10 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.139+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68014660 con 0x7f9b780cdc10 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.139+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b780cc460 con 0x7f9b780cdc10 2024-03-29T04:09:22.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.140+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b780a2480 con 0x7f9b780cdc10 2024-03-29T04:09:22.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.141+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9b68012040 con 0x7f9b780cdc10 2024-03-29T04:09:22.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.141+0000 7f9b6e7fc640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9b5c06c560 0x7f9b5c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.141+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9b6808baf0 con 0x7f9b780cdc10 2024-03-29T04:09:22.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.141+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9b5c06c560 0x7f9b5c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.142+0000 7f9b7dcc0640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9b5c06c560 0x7f9b5c06ea20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9b74033000 tx=0x7f9b74009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.145+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9b780cbf00 con 0x7f9b780cdc10 2024-03-29T04:09:22.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.327+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9b780d1d80 con 0x7f9b780cdc10 2024-03-29T04:09:22.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.327+0000 7f9b6e7fc640 1 -- 172.21.15.156:0/3287465331 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f9b68019030 con 0x7f9b780cdc10 2024-03-29T04:09:22.307 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9b5c06c560 msgr2=0x7f9b5c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9b5c06c560 0x7f9b5c06ea20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9b74033000 tx=0x7f9b74009030 comp rx=0 tx=0).stop 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 msgr2=0x7f9b78158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 0x7f9b78158e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9b68011aa0 tx=0x7f9b68011f60 comp rx=0 tx=0).stop 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 shutdown_connections 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9b5c06c560 0x7f9b5c06ea20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b781595f0 0x7f9b78160c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b780cdc10 0x7f9b78158e10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 --2- 172.21.15.156:0/3287465331 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b640039b0 0x7f9b780ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 >> 172.21.15.156:0/3287465331 conn(0x7f9b7809d7a0 msgr2=0x7f9b780cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 shutdown_connections 2024-03-29T04:09:22.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.329+0000 7f9b7ff24640 1 -- 172.21.15.156:0/3287465331 wait complete. 2024-03-29T04:09:22.320 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:22.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.446+0000 7fa5e0f52640 1 Processor -- start 2024-03-29T04:09:22.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.446+0000 7fa5e0f52640 1 -- start start 2024-03-29T04:09:22.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5e0f52640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa5dc0d43e0 0x7fa5dc0d47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5e0f52640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa5dc0cfa70 0x7fa5dc0cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5e0f52640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5dc0ceff0 con 0x7fa5dc0c7fa0 2024-03-29T04:09:22.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5e0f52640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa5dc0cf5e0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5e0f52640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa5dc0d4d00 con 0x7fa5dc0cfa70 2024-03-29T04:09:22.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5da59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa5dc0d43e0 0x7fa5dc0d47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5d3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa5dc0c7fa0 0x7fa5dc0b43a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57950/0 (socket says 172.21.15.156:57950) 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5d3fff640 1 -- 172.21.15.156:0/218381084 learned_addr learned my addr 172.21.15.156:0/218381084 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5dad9d640 1 --2- 172.21.15.156:0/218381084 >> v2:172.21.15.203:3300/0 conn(0x7fa5dc0cfa70 0x7fa5dc0cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5da59c640 1 -- 172.21.15.156:0/218381084 >> v2:172.21.15.203:3300/0 conn(0x7fa5dc0cfa70 msgr2=0x7fa5dc0cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5da59c640 1 --2- 172.21.15.156:0/218381084 >> v2:172.21.15.203:3300/0 conn(0x7fa5dc0cfa70 0x7fa5dc0cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5da59c640 1 -- 172.21.15.156:0/218381084 >> v1:172.21.15.156:6789/0 conn(0x7fa5dc0c7fa0 legacy=0x7fa5dc0b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.447+0000 7fa5d9d9b640 1 -- 172.21.15.156:0/218381084 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 568172892 0 0) 0x7fa5c4001110 con 0x7fa5dc0c7fa0 2024-03-29T04:09:22.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.448+0000 7fa5da59c640 1 -- 172.21.15.156:0/218381084 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5dc0b4f10 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.448+0000 7fa5da59c640 1 --2- 172.21.15.156:0/218381084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc0d47c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0d5260 tx=0x7fa5cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7690193f02944b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.448+0000 7fa5d9d9b640 1 -- 172.21.15.156:0/218381084 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc035020 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.448+0000 7fa5d9d9b640 1 -- 172.21.15.156:0/218381084 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5cc031830 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.448+0000 7fa5d9d9b640 1 -- 172.21.15.156:0/218381084 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc031c00 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 -- 172.21.15.156:0/218381084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 msgr2=0x7fa5dc0d47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/218381084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc0d47c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0d5260 tx=0x7fa5cc02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 -- 172.21.15.156:0/218381084 shutdown_connections 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/218381084 >> v2:172.21.15.203:3300/0 conn(0x7fa5dc0cfa70 0x7fa5dc0cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/218381084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc0d47c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 -- 172.21.15.156:0/218381084 >> 172.21.15.156:0/218381084 conn(0x7fa5dc08bfd0 msgr2=0x7fa5dc09d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 -- 172.21.15.156:0/218381084 shutdown_connections 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.449+0000 7fa5e0f52640 1 -- 172.21.15.156:0/218381084 wait complete. 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.450+0000 7fa5e0f52640 1 Processor -- start 2024-03-29T04:09:22.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.450+0000 7fa5e0f52640 1 -- start start 2024-03-29T04:09:22.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5e0f52640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5dc0c7fa0 0x7fa5dc151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5e0f52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5dc0cfa70 0x7fa5dc0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5e0f52640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5e0f52640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc0d22e0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5e0f52640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc0cf270 con 0x7fa5dc0cfa70 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5e0f52640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa5dc15ea70 con 0x7fa5dc0c7fa0 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc160f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42632/0 (socket says 172.21.15.156:42632) 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 -- 172.21.15.156:0/3872962465 learned_addr learned my addr 172.21.15.156:0/3872962465 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5d3fff640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5dc0cfa70 0x7fa5dc0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5da59c640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5dc0c7fa0 0x7fa5dc151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 -- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5dc0c7fa0 msgr2=0x7fa5dc151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5dc0c7fa0 0x7fa5dc151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 -- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5dc0cfa70 msgr2=0x7fa5dc0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5dc0cfa70 0x7fa5dc0cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5dad9d640 1 -- 172.21.15.156:0/3872962465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5dc0cbfd0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5d3fff640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5dc0cfa70 0x7fa5dc0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.451+0000 7fa5da59c640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5dc0c7fa0 0x7fa5dc151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.452+0000 7fa5dad9d640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc160f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4007e40 tx=0x7fa5d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.452+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4014620 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.452+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d4019070 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.452+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5dc151e50 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.452+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4014a20 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.452+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5dc0cc270 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.453+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5dc08cec0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.454+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa5d4014bc0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.454+0000 7fa5d2ffd640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5bc06c530 0x7fa5bc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.454+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa5d408bfe0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.454+0000 7fa5da59c640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5bc06c530 0x7fa5bc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.455+0000 7fa5da59c640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5bc06c530 0x7fa5bc06e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc036000 tx=0x7fa5cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.463+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa5dc08cec0 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.642+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa5dc0a2380 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.642+0000 7fa5d2ffd640 1 -- 172.21.15.156:0/3872962465 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fa5d405a600 con 0x7fa5dc0d43e0 2024-03-29T04:09:22.622 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:22.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5bc06c530 msgr2=0x7fa5bc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5bc06c530 0x7fa5bc06e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc036000 tx=0x7fa5cc008030 comp rx=0 tx=0).stop 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 msgr2=0x7fa5dc160f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc160f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4007e40 tx=0x7fa5d400e4b0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 shutdown_connections 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5bc06c530 0x7fa5bc06e9f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5dc0d43e0 0x7fa5dc160f20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5dc0cfa70 0x7fa5dc0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 --2- 172.21.15.156:0/3872962465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5dc0c7fa0 0x7fa5dc151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.644+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 >> 172.21.15.156:0/3872962465 conn(0x7fa5dc08bfd0 msgr2=0x7fa5dc06d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.645+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 shutdown_connections 2024-03-29T04:09:22.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.645+0000 7fa5e0f52640 1 -- 172.21.15.156:0/3872962465 wait complete. 2024-03-29T04:09:22.641 INFO:tasks.cephfs_test_runner:Starting test: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2024-03-29T04:09:22.642 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:22.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.763+0000 7fd6b8324640 1 Processor -- start 2024-03-29T04:09:22.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.763+0000 7fd6b8324640 1 -- start start 2024-03-29T04:09:22.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b8324640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd6b00b1850 0x7fd6b00b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b8324640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd6b00b32e0 0x7fd6b009aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b8324640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6b00b1d00 con 0x7fd6b015fba0 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b8324640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd6b00b1fe0 con 0x7fd6b00b1850 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b8324640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd6b00b22c0 con 0x7fd6b00b32e0 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b58bf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd6b00b1850 0x7fd6b00b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b58bf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd6b00b1850 0x7fd6b00b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42636/0 (socket says 172.21.15.156:42636) 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.764+0000 7fd6b58bf640 1 -- 172.21.15.156:0/2684399425 learned_addr learned my addr 172.21.15.156:0/2684399425 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.765+0000 7fd6b68c1640 1 --2- 172.21.15.156:0/2684399425 >> v2:172.21.15.203:3300/0 conn(0x7fd6b00b32e0 0x7fd6b009aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.765+0000 7fd6b58bf640 1 -- 172.21.15.156:0/2684399425 >> v2:172.21.15.203:3300/0 conn(0x7fd6b00b32e0 msgr2=0x7fd6b009aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.765+0000 7fd6b58bf640 1 --2- 172.21.15.156:0/2684399425 >> v2:172.21.15.203:3300/0 conn(0x7fd6b00b32e0 0x7fd6b009aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.765+0000 7fd6b58bf640 1 -- 172.21.15.156:0/2684399425 >> v1:172.21.15.203:6789/0 conn(0x7fd6b015fba0 legacy=0x7fd6b0160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.765+0000 7fd6b58bf640 1 -- 172.21.15.156:0/2684399425 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b00b25a0 con 0x7fd6b00b1850 2024-03-29T04:09:22.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.766+0000 7fd6b58bf640 1 --2- 172.21.15.156:0/2684399425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00b2da0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00d1280 tx=0x7fd6a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63bd8a44528c7235 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.766+0000 7fd6b50be640 1 -- 172.21.15.156:0/2684399425 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0035020 con 0x7fd6b00b1850 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.766+0000 7fd6b50be640 1 -- 172.21.15.156:0/2684399425 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a0031870 con 0x7fd6b00b1850 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.766+0000 7fd6b50be640 1 -- 172.21.15.156:0/2684399425 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0031c40 con 0x7fd6b00b1850 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 -- 172.21.15.156:0/2684399425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 msgr2=0x7fd6b00b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2684399425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00b2da0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00d1280 tx=0x7fd6a002b910 comp rx=0 tx=0).stop 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 -- 172.21.15.156:0/2684399425 shutdown_connections 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2684399425 >> v2:172.21.15.203:3300/0 conn(0x7fd6b00b32e0 0x7fd6b009aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2684399425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00b2da0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 -- 172.21.15.156:0/2684399425 >> 172.21.15.156:0/2684399425 conn(0x7fd6b00b3d90 msgr2=0x7fd6b00b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 -- 172.21.15.156:0/2684399425 shutdown_connections 2024-03-29T04:09:22.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.767+0000 7fd6b8324640 1 -- 172.21.15.156:0/2684399425 wait complete. 2024-03-29T04:09:22.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.768+0000 7fd6b8324640 1 Processor -- start 2024-03-29T04:09:22.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.768+0000 7fd6b8324640 1 -- start start 2024-03-29T04:09:22.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd6b00b32e0 0x7fd6b00c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6b015fba0 0x7fd6b00c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b009af80 con 0x7fd6b00b1850 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b00b3990 con 0x7fd6b015fba0 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b009b100 con 0x7fd6b00b32e0 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42642/0 (socket says 172.21.15.156:42642) 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 -- 172.21.15.156:0/2347644377 learned_addr learned my addr 172.21.15.156:0/2347644377 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b58bf640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd6b00b32e0 0x7fd6b00c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b68c1640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6b015fba0 0x7fd6b00c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 -- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd6b00b32e0 msgr2=0x7fd6b00c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd6b00b32e0 0x7fd6b00c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 -- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6b015fba0 msgr2=0x7fd6b00c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6b015fba0 0x7fd6b00c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 -- 172.21.15.156:0/2347644377 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b015e650 con 0x7fd6b00b1850 2024-03-29T04:09:22.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b60c0640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00c8030 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00c3f0 tx=0x7fd6ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0052d0 con 0x7fd6b00b1850 2024-03-29T04:09:22.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ac005490 con 0x7fd6b00b1850 2024-03-29T04:09:22.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b015e930 con 0x7fd6b00b1850 2024-03-29T04:09:22.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.769+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac007260 con 0x7fd6b00b1850 2024-03-29T04:09:22.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.770+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b00c8570 con 0x7fd6b00b1850 2024-03-29T04:09:22.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.771+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b0068d20 con 0x7fd6b00b1850 2024-03-29T04:09:22.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.771+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd6ac007400 con 0x7fd6b00b1850 2024-03-29T04:09:22.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.771+0000 7fd6a6ffd640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd69406c530 0x7fd69406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:22.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.772+0000 7fd6b58bf640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd69406c530 0x7fd69406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:22.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.772+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd6ac08ab60 con 0x7fd6b00b1850 2024-03-29T04:09:22.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.772+0000 7fd6b58bf640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd69406c530 0x7fd69406e9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0036000 tx=0x7fd6a003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:22.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.776+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd6ac059210 con 0x7fd6b00b1850 2024-03-29T04:09:22.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.959+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6b0073bf0 con 0x7fd6b00b1850 2024-03-29T04:09:22.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.960+0000 7fd6a6ffd640 1 -- 172.21.15.156:0/2347644377 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fd6ac058b10 con 0x7fd6b00b1850 2024-03-29T04:09:22.940 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:22.940 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd69406c530 msgr2=0x7fd69406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd69406c530 0x7fd69406e9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0036000 tx=0x7fd6a003c040 comp rx=0 tx=0).stop 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 msgr2=0x7fd6b00c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00c8030 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00c3f0 tx=0x7fd6ac00c8b0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 shutdown_connections 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd69406c530 0x7fd69406e9f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6b015fba0 0x7fd6b00c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd6b00b32e0 0x7fd6b00c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 --2- 172.21.15.156:0/2347644377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6b00b1850 0x7fd6b00c8030 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 >> 172.21.15.156:0/2347644377 conn(0x7fd6b00b3d90 msgr2=0x7fd6b015acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 shutdown_connections 2024-03-29T04:09:22.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:22.962+0000 7fd6b8324640 1 -- 172.21.15.156:0/2347644377 wait complete. 2024-03-29T04:09:22.953 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:22.953 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:23.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.081+0000 7fa22ca9a640 1 Processor -- start 2024-03-29T04:09:23.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.082+0000 7fa22ca9a640 1 -- start start 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22ca9a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa22815e920 0x7fa2280d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22ca9a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa2280cfc30 0x7fa2280d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22ca9a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2280cf190 con 0x7fa2280d2120 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22ca9a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa2280cf7a0 con 0x7fa22815e920 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22ca9a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa2280d3080 con 0x7fa2280cfc30 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa22815e920 0x7fa2280d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa22815e920 0x7fa2280d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42646/0 (socket says 172.21.15.156:42646) 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa225d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa2280d2120 0x7fa22806c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57966/0 (socket says 172.21.15.156:57966) 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22659c640 1 -- 172.21.15.156:0/3239611061 learned_addr learned my addr 172.21.15.156:0/3239611061 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa226d9d640 1 --2- 172.21.15.156:0/3239611061 >> v2:172.21.15.203:3300/0 conn(0x7fa2280cfc30 0x7fa2280d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22659c640 1 -- 172.21.15.156:0/3239611061 >> v2:172.21.15.203:3300/0 conn(0x7fa2280cfc30 msgr2=0x7fa2280d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22659c640 1 --2- 172.21.15.156:0/3239611061 >> v2:172.21.15.203:3300/0 conn(0x7fa2280cfc30 0x7fa2280d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.083+0000 7fa22659c640 1 -- 172.21.15.156:0/3239611061 >> v1:172.21.15.156:6789/0 conn(0x7fa2280d2120 legacy=0x7fa22806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.084+0000 7fa22559a640 1 -- 172.21.15.156:0/3239611061 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612895715 0 0) 0x7fa2280cf190 con 0x7fa2280d2120 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.084+0000 7fa22659c640 1 -- 172.21.15.156:0/3239611061 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2280cf430 con 0x7fa22815e920 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.084+0000 7fa22659c640 1 --2- 172.21.15.156:0/3239611061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280d26e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa22808ca00 tx=0x7fa22002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18be40c782252b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.084+0000 7fa22559a640 1 -- 172.21.15.156:0/3239611061 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22002cb20 con 0x7fa22815e920 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.084+0000 7fa22559a640 1 -- 172.21.15.156:0/3239611061 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa22002ccc0 con 0x7fa22815e920 2024-03-29T04:09:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.084+0000 7fa22559a640 1 -- 172.21.15.156:0/3239611061 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220034610 con 0x7fa22815e920 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 -- 172.21.15.156:0/3239611061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 msgr2=0x7fa2280d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/3239611061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280d26e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa22808ca00 tx=0x7fa22002bf80 comp rx=0 tx=0).stop 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 -- 172.21.15.156:0/3239611061 shutdown_connections 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/3239611061 >> v2:172.21.15.203:3300/0 conn(0x7fa2280cfc30 0x7fa2280d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/3239611061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280d26e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 -- 172.21.15.156:0/3239611061 >> 172.21.15.156:0/3239611061 conn(0x7fa22809d7a0 msgr2=0x7fa2280d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 -- 172.21.15.156:0/3239611061 shutdown_connections 2024-03-29T04:09:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.085+0000 7fa22ca9a640 1 -- 172.21.15.156:0/3239611061 wait complete. 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.086+0000 7fa22ca9a640 1 Processor -- start 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.086+0000 7fa22ca9a640 1 -- start start 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22ca9a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2280cfc30 0x7fa22809ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22ca9a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2280d2120 0x7fa2280b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22ca9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22ca9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2280d34e0 con 0x7fa22815e920 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22ca9a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2280d3360 con 0x7fa2280d2120 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22ca9a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa2280d2c20 con 0x7fa2280cfc30 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa226d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa226d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42650/0 (socket says 172.21.15.156:42650) 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa226d9d640 1 -- 172.21.15.156:0/2147196987 learned_addr learned my addr 172.21.15.156:0/2147196987 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa225d9b640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2280d2120 0x7fa2280b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.087+0000 7fa22659c640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2280cfc30 0x7fa22809ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa226d9d640 1 -- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2280cfc30 msgr2=0x7fa22809ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa226d9d640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2280cfc30 0x7fa22809ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa226d9d640 1 -- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2280d2120 msgr2=0x7fa2280b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa226d9d640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2280d2120 0x7fa2280b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa226d9d640 1 -- 172.21.15.156:0/2147196987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2280b45b0 con 0x7fa22815e920 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa226d9d640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280cd960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa21800d800 tx=0x7fa21800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21800b6c0 con 0x7fa22815e920 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa218019070 con 0x7fa22815e920 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22814d480 con 0x7fa22815e920 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21800bac0 con 0x7fa22815e920 2024-03-29T04:09:23.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.088+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa22814da40 con 0x7fa22815e920 2024-03-29T04:09:23.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.089+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2280a2480 con 0x7fa22815e920 2024-03-29T04:09:23.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.090+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa21800bc60 con 0x7fa22815e920 2024-03-29T04:09:23.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.090+0000 7fa216ffd640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa20806c530 0x7fa20806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.090+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa21808b9e0 con 0x7fa22815e920 2024-03-29T04:09:23.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.090+0000 7fa22659c640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa20806c530 0x7fa20806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.091+0000 7fa22659c640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa20806c530 0x7fa20806e9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa220035000 tx=0x7fa220030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.095+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa2280b45b0 con 0x7fa22815e920 2024-03-29T04:09:23.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.276+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa22804f9d0 con 0x7fa22815e920 2024-03-29T04:09:23.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.276+0000 7fa216ffd640 1 -- 172.21.15.156:0/2147196987 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fa2180599b0 con 0x7fa22815e920 2024-03-29T04:09:23.256 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa20806c530 msgr2=0x7fa20806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa20806c530 0x7fa20806e9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa220035000 tx=0x7fa220030000 comp rx=0 tx=0).stop 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 msgr2=0x7fa2280cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280cd960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa21800d800 tx=0x7fa21800dcc0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 shutdown_connections 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa20806c530 0x7fa20806e9f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa22815e920 0x7fa2280cd960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2280d2120 0x7fa2280b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 --2- 172.21.15.156:0/2147196987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2280cfc30 0x7fa22809ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 >> 172.21.15.156:0/2147196987 conn(0x7fa22809d7a0 msgr2=0x7fa22806c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:23.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 shutdown_connections 2024-03-29T04:09:23.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.278+0000 7fa22ca9a640 1 -- 172.21.15.156:0/2147196987 wait complete. 2024-03-29T04:09:23.269 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:23.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.394+0000 7f8dd0be2640 1 Processor -- start 2024-03-29T04:09:23.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.394+0000 7f8dd0be2640 1 -- start start 2024-03-29T04:09:23.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dd0be2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8dcc159160 0x7f8dcc0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dd0be2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8dcc0d1850 0x7f8dcc164fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dd0be2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dcc0b5630 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dd0be2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8dcc0b4190 con 0x7f8dcc159160 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dd0be2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8dcc0af170 con 0x7f8dcc0d1850 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dca59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8dcc159160 0x7f8dcc0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dc3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8dcc0aebf0 0x7f8dcc0d1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57972/0 (socket says 172.21.15.156:57972) 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.395+0000 7f8dc3fff640 1 -- 172.21.15.156:0/1261277414 learned_addr learned my addr 172.21.15.156:0/1261277414 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.396+0000 7f8dcad9d640 1 --2- 172.21.15.156:0/1261277414 >> v2:172.21.15.203:3300/0 conn(0x7f8dcc0d1850 0x7f8dcc164fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.396+0000 7f8dca59c640 1 -- 172.21.15.156:0/1261277414 >> v2:172.21.15.203:3300/0 conn(0x7f8dcc0d1850 msgr2=0x7f8dcc164fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.396+0000 7f8dca59c640 1 --2- 172.21.15.156:0/1261277414 >> v2:172.21.15.203:3300/0 conn(0x7f8dcc0d1850 0x7f8dcc164fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.396+0000 7f8dca59c640 1 -- 172.21.15.156:0/1261277414 >> v1:172.21.15.156:6789/0 conn(0x7f8dcc0aebf0 legacy=0x7f8dcc0d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.396+0000 7f8dca59c640 1 -- 172.21.15.156:0/1261277414 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc0af450 con 0x7f8dcc159160 2024-03-29T04:09:23.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.396+0000 7f8dc9d9b640 1 -- 172.21.15.156:0/1261277414 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130928707 0 0) 0x7f8dcc0b5630 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.397+0000 7f8dca59c640 1 --2- 172.21.15.156:0/1261277414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc159160 0x7f8dcc0ae6b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc15eef0 tx=0x7f8dbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90fdc059c2f76c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.397+0000 7f8dc9d9b640 1 -- 172.21.15.156:0/1261277414 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc035020 con 0x7f8dcc159160 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.397+0000 7f8dc9d9b640 1 -- 172.21.15.156:0/1261277414 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dbc031830 con 0x7f8dcc159160 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.397+0000 7f8dc9d9b640 1 -- 172.21.15.156:0/1261277414 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc031c00 con 0x7f8dcc159160 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 -- 172.21.15.156:0/1261277414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc159160 msgr2=0x7f8dcc0ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/1261277414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc159160 0x7f8dcc0ae6b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc15eef0 tx=0x7f8dbc02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 -- 172.21.15.156:0/1261277414 shutdown_connections 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/1261277414 >> v2:172.21.15.203:3300/0 conn(0x7f8dcc0d1850 0x7f8dcc164fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/1261277414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc159160 0x7f8dcc0ae6b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 -- 172.21.15.156:0/1261277414 >> 172.21.15.156:0/1261277414 conn(0x7f8dcc0b6580 msgr2=0x7f8dcc0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:23.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 -- 172.21.15.156:0/1261277414 shutdown_connections 2024-03-29T04:09:23.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.398+0000 7f8dd0be2640 1 -- 172.21.15.156:0/1261277414 wait complete. 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.399+0000 7f8dd0be2640 1 Processor -- start 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.399+0000 7f8dd0be2640 1 -- start start 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dd0be2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 0x7f8dcc1561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dd0be2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8dcc0d1850 0x7f8dcc153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dd0be2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8dcc150f90 0x7f8dcc0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dd0be2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc15cad0 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dd0be2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0afa10 con 0x7f8dcc150f90 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dd0be2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8dcc15bb80 con 0x7f8dcc0d1850 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 0x7f8dcc1561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 0x7f8dcc1561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42670/0 (socket says 172.21.15.156:42670) 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 -- 172.21.15.156:0/3009935356 learned_addr learned my addr 172.21.15.156:0/3009935356 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:23.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dc3fff640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8dcc0d1850 0x7f8dcc153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dcad9d640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8dcc150f90 0x7f8dcc0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 -- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8dcc0d1850 msgr2=0x7f8dcc153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8dcc0d1850 0x7f8dcc153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 -- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8dcc150f90 msgr2=0x7f8dcc0cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8dcc150f90 0x7f8dcc0cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 -- 172.21.15.156:0/3009935356 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc154240 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dcad9d640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8dcc150f90 0x7f8dcc0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dc3fff640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8dcc0d1850 0x7f8dcc153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dca59c640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 0x7f8dcc1561c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc000c00 tx=0x7f8dbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.400+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc03a9b0 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.401+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dbc03ab70 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.401+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dcc153c80 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.401+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc0443f0 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.401+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dcc151680 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.402+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dcc09f3c0 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.402+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8dbc038020 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.403+0000 7f8dc2ffd640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8dac06c560 0x7f8dac06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.403+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8dbc0ad2b0 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.403+0000 7f8dc3fff640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8dac06c560 0x7f8dac06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.403+0000 7f8dc3fff640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8dac06c560 0x7f8dac06ea20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8db4007fd0 tx=0x7f8db4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.411+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8dcc154240 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.591+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8dcc04f9d0 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.592+0000 7f8dc2ffd640 1 -- 172.21.15.156:0/3009935356 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f8dbc03f070 con 0x7f8dcc0aebf0 2024-03-29T04:09:23.571 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:23.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.593+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8dac06c560 msgr2=0x7f8dac06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.593+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8dac06c560 0x7f8dac06ea20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8db4007fd0 tx=0x7f8db4009040 comp rx=0 tx=0).stop 2024-03-29T04:09:23.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.593+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 msgr2=0x7f8dcc1561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.593+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 0x7f8dcc1561c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc000c00 tx=0x7f8dbc0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 shutdown_connections 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8dac06c560 0x7f8dac06ea20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8dcc150f90 0x7f8dcc0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8dcc0d1850 0x7f8dcc153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 --2- 172.21.15.156:0/3009935356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc0aebf0 0x7f8dcc1561c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 >> 172.21.15.156:0/3009935356 conn(0x7f8dcc0b6580 msgr2=0x7f8dcc15a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 shutdown_connections 2024-03-29T04:09:23.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.594+0000 7f8dd0be2640 1 -- 172.21.15.156:0/3009935356 wait complete. 2024-03-29T04:09:23.584 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:23.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.710+0000 7f4696c93640 1 Processor -- start 2024-03-29T04:09:23.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.710+0000 7f4696c93640 1 -- start start 2024-03-29T04:09:23.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.711+0000 7f4696c93640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4690159160 0x7f46900ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.711+0000 7f4696c93640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 0x7f4690164fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.711+0000 7f4696c93640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46900b5630 con 0x7f46900aebf0 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.711+0000 7f4696c93640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f46900b4190 con 0x7f4690159160 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.711+0000 7f4696c93640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f46900af170 con 0x7f46900d1850 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4695230640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 0x7f4690164fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4690159160 0x7f46900ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4690159160 0x7f46900ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:44806/0 (socket says 172.21.15.156:44806) 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4695230640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 0x7f4690164fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:40796/0 (socket says 172.21.15.156:40796) 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f468ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f46900aebf0 0x7f46900d1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:55382/0 (socket says 172.21.15.156:55382) 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 -- 172.21.15.156:0/4161411039 learned_addr learned my addr 172.21.15.156:0/4161411039 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 -- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 msgr2=0x7f4690164fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 --2- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 0x7f4690164fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 -- 172.21.15.156:0/4161411039 >> v1:172.21.15.203:6789/0 conn(0x7f46900aebf0 legacy=0x7f46900d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4694a2f640 1 -- 172.21.15.156:0/4161411039 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46900af450 con 0x7f4690159160 2024-03-29T04:09:23.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.712+0000 7f4695230640 1 --2- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 0x7f4690164fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.713+0000 7f4694a2f640 1 --2- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4690159160 0x7f46900ae6b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4684009870 tx=0x7f468402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7278f2087ad20d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.713+0000 7f468f7fe640 1 -- 172.21.15.156:0/4161411039 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684034070 con 0x7f4690159160 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.713+0000 7f468f7fe640 1 -- 172.21.15.156:0/4161411039 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4684030840 con 0x7f4690159160 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.713+0000 7f468f7fe640 1 -- 172.21.15.156:0/4161411039 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684030c10 con 0x7f4690159160 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 -- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4690159160 msgr2=0x7f46900ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 --2- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4690159160 0x7f46900ae6b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4684009870 tx=0x7f468402b980 comp rx=0 tx=0).stop 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 -- 172.21.15.156:0/4161411039 shutdown_connections 2024-03-29T04:09:23.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 --2- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900d1850 0x7f4690164fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 --2- 172.21.15.156:0/4161411039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4690159160 0x7f46900ae6b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 -- 172.21.15.156:0/4161411039 >> 172.21.15.156:0/4161411039 conn(0x7f46900b6580 msgr2=0x7f46900b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:23.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 -- 172.21.15.156:0/4161411039 shutdown_connections 2024-03-29T04:09:23.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.714+0000 7f4696c93640 1 -- 172.21.15.156:0/4161411039 wait complete. 2024-03-29T04:09:23.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.715+0000 7f4696c93640 1 Processor -- start 2024-03-29T04:09:23.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.715+0000 7f4696c93640 1 -- start start 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4696c93640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900aebf0 0x7f469014bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4696c93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46900d1850 0x7f469006f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4696c93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 0x7f46900afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4696c93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f469015cad0 con 0x7f469006fd80 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4696c93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46900afa10 con 0x7f46900d1850 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4696c93640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f469015bb80 con 0x7f46900aebf0 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 0x7f46900afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 0x7f46900afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42676/0 (socket says 172.21.15.156:42676) 2024-03-29T04:09:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 -- 172.21.15.156:0/1856972932 learned_addr learned my addr 172.21.15.156:0/1856972932 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f468ffff640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46900d1850 0x7f469006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4694a2f640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900aebf0 0x7f469014bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 -- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900aebf0 msgr2=0x7f469014bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900aebf0 0x7f469014bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 -- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46900d1850 msgr2=0x7f469006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46900d1850 0x7f469006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 -- 172.21.15.156:0/1856972932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46900cf1f0 con 0x7f469006fd80 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.716+0000 7f4695230640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 0x7f46900afc20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f468800efc0 tx=0x7f468800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.717+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688018020 con 0x7f469006fd80 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.717+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4688014480 con 0x7f469006fd80 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.717+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4690155570 con 0x7f469006fd80 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.717+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688014850 con 0x7f469006fd80 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.717+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46900b0c30 con 0x7f469006fd80 2024-03-29T04:09:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.718+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469009f3c0 con 0x7f469006fd80 2024-03-29T04:09:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.718+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4688005d10 con 0x7f469006fd80 2024-03-29T04:09:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.719+0000 7f468d7fa640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f467406c530 0x7f467406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.719+0000 7f4694a2f640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f467406c530 0x7f467406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.719+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f468808c3e0 con 0x7f469006fd80 2024-03-29T04:09:23.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.719+0000 7f4694a2f640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f467406c530 0x7f467406e9f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4684036000 tx=0x7f468403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:23.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.723+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4688022030 con 0x7f469006fd80 2024-03-29T04:09:23.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.905+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4690158980 con 0x7f469006fd80 2024-03-29T04:09:23.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.906+0000 7f468d7fa640 1 -- 172.21.15.156:0/1856972932 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f4688010040 con 0x7f469006fd80 2024-03-29T04:09:23.885 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:23.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f467406c530 msgr2=0x7f467406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f467406c530 0x7f467406e9f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4684036000 tx=0x7f468403b040 comp rx=0 tx=0).stop 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 msgr2=0x7f46900afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 0x7f46900afc20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f468800efc0 tx=0x7f468800c490 comp rx=0 tx=0).stop 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 shutdown_connections 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f467406c530 0x7f467406e9f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f469006fd80 0x7f46900afc20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46900d1850 0x7f469006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 --2- 172.21.15.156:0/1856972932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46900aebf0 0x7f469014bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 >> 172.21.15.156:0/1856972932 conn(0x7f46900b6580 msgr2=0x7f4690159550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 shutdown_connections 2024-03-29T04:09:23.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:23.908+0000 7f4696c93640 1 -- 172.21.15.156:0/1856972932 wait complete. 2024-03-29T04:09:23.899 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_uninlining.TestDataUninlining.test_data_uninlining' 2024-03-29T04:09:24.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.025+0000 7f7fdad0d640 1 Processor -- start 2024-03-29T04:09:24.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.025+0000 7f7fdad0d640 1 -- start start 2024-03-29T04:09:24.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fdad0d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd415b460 0x7f7fd40ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fdad0d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7fd4071810 0x7f7fd4071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fdad0d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7fd40d0470 0x7f7fd40d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fdad0d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd40729b0 con 0x7f7fd4071810 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fdad0d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd4072c90 con 0x7f7fd40d0470 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fdad0d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd4072f70 con 0x7f7fd415b460 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fd3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7fd4071810 0x7f7fd4071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fd3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7fd4071810 0x7f7fd4071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42678/0 (socket says 172.21.15.156:42678) 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.026+0000 7f7fd3fff640 1 -- 172.21.15.156:0/91467856 learned_addr learned my addr 172.21.15.156:0/91467856 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd92aa640 1 --2- 172.21.15.156:0/91467856 >> v2:172.21.15.203:3300/0 conn(0x7f7fd40d0470 0x7f7fd40d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd8aa9640 1 --2- 172.21.15.156:0/91467856 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd415b460 0x7f7fd40ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd3fff640 1 -- 172.21.15.156:0/91467856 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd415b460 msgr2=0x7f7fd40ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd3fff640 1 --2- 172.21.15.156:0/91467856 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd415b460 0x7f7fd40ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd3fff640 1 -- 172.21.15.156:0/91467856 >> v2:172.21.15.203:3300/0 conn(0x7f7fd40d0470 msgr2=0x7f7fd40d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:24.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd3fff640 1 --2- 172.21.15.156:0/91467856 >> v2:172.21.15.203:3300/0 conn(0x7f7fd40d0470 0x7f7fd40d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.027+0000 7f7fd3fff640 1 -- 172.21.15.156:0/91467856 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd4072680 con 0x7f7fd4071810 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.028+0000 7f7fd3fff640 1 --2- 172.21.15.156:0/91467856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd4071810 0x7f7fd4071c90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7fd406f7c0 tx=0x7f7fbc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a44867c88df324 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.028+0000 7f7fd37fe640 1 -- 172.21.15.156:0/91467856 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc038070 con 0x7f7fd4071810 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.028+0000 7f7fd37fe640 1 -- 172.21.15.156:0/91467856 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fbc02f050 con 0x7f7fd4071810 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.028+0000 7f7fd37fe640 1 -- 172.21.15.156:0/91467856 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc031970 con 0x7f7fd4071810 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 -- 172.21.15.156:0/91467856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd4071810 msgr2=0x7f7fd4071c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/91467856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd4071810 0x7f7fd4071c90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7fd406f7c0 tx=0x7f7fbc02b7f0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 -- 172.21.15.156:0/91467856 shutdown_connections 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/91467856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd4071810 0x7f7fd4071c90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/91467856 >> v2:172.21.15.203:3300/0 conn(0x7f7fd40d0470 0x7f7fd40d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/91467856 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd415b460 0x7f7fd40ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 -- 172.21.15.156:0/91467856 >> 172.21.15.156:0/91467856 conn(0x7f7fd40b21e0 msgr2=0x7f7fd40b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 -- 172.21.15.156:0/91467856 shutdown_connections 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.029+0000 7f7fdad0d640 1 -- 172.21.15.156:0/91467856 wait complete. 2024-03-29T04:09:24.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.030+0000 7f7fdad0d640 1 Processor -- start 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.030+0000 7f7fdad0d640 1 -- start start 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fdad0d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd4071810 0x7f7fd4152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fdad0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd40d0470 0x7f7fd41598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fdad0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 0x7f7fd4159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fdad0d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd40b37c0 con 0x7f7fd415b460 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fdad0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd40b3640 con 0x7f7fd40d0470 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fdad0d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd40b3940 con 0x7f7fd4071810 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 0x7f7fd4159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 0x7f7fd4159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42682/0 (socket says 172.21.15.156:42682) 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 -- 172.21.15.156:0/531106390 learned_addr learned my addr 172.21.15.156:0/531106390 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd3fff640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd40d0470 0x7f7fd41598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd8aa9640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd4071810 0x7f7fd4152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 -- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd4071810 msgr2=0x7f7fd4152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd4071810 0x7f7fd4152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 -- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd40d0470 msgr2=0x7f7fd41598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd40d0470 0x7f7fd41598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 -- 172.21.15.156:0/531106390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd41464c0 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd92aa640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 0x7f7fd4159e20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00ef00 tx=0x7f7fcc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc018020 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.031+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc014400 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.032+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc0147f0 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.032+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd41467a0 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.032+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd4146d60 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.033+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd40b44b0 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.033+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7fcc005d10 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.033+0000 7f7fd17fa640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fb406c530 0x7f7fb406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.033+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7fcc08bdd0 con 0x7f7fd415b460 2024-03-29T04:09:24.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.033+0000 7f7fd8aa9640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fb406c530 0x7f7fb406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:24.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.034+0000 7f7fd8aa9640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fb406c530 0x7f7fb406e9f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009a30 tx=0x7f7fc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:24.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.038+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7fcc022070 con 0x7f7fd415b460 2024-03-29T04:09:24.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.452+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_uninlining.TestDataUninlining.test_data_uninlining"]} v 0) -- 0x7f7fd4068610 con 0x7f7fd415b460 2024-03-29T04:09:24.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.904+0000 7f7fd17fa640 1 -- 172.21.15.156:0/531106390 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_uninlining.TestDataUninlining.test_data_uninlining"]}]=0 v55) ==== 150+0+0 (secure 0 0 0) 0x7f7fcc08d0d0 con 0x7f7fd415b460 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.906+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fb406c530 msgr2=0x7f7fb406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.906+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fb406c530 0x7f7fb406e9f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009a30 tx=0x7f7fc4008040 comp rx=0 tx=0).stop 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.906+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 msgr2=0x7f7fd4159e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.906+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 0x7f7fd4159e20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00ef00 tx=0x7f7fcc00b480 comp rx=0 tx=0).stop 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 shutdown_connections 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7fb406c530 0x7f7fb406e9f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fd415b460 0x7f7fd4159e20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd40d0470 0x7f7fd41598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 --2- 172.21.15.156:0/531106390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fd4071810 0x7f7fd4152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:24.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 >> 172.21.15.156:0/531106390 conn(0x7f7fd40b21e0 msgr2=0x7f7fd40b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:24.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 shutdown_connections 2024-03-29T04:09:24.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:24.907+0000 7f7fdad0d640 1 -- 172.21.15.156:0/531106390 wait complete. 2024-03-29T04:09:24.902 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T04:09:25.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.031+0000 7f838ade2640 1 Processor -- start 2024-03-29T04:09:25.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.031+0000 7f838ade2640 1 -- start start 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838ade2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f83840d2120 0x7f838406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838ade2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f838406cb10 0x7f83840cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838ade2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83840d2df0 con 0x7f838415e920 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838ade2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f83840cf7a0 con 0x7f838406cb10 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838ade2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f83840cf190 con 0x7f83840d2120 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838937f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f838406cb10 0x7f83840cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f8388b7e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f838415e920 0x7f83840d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57980/0 (socket says 172.21.15.156:57980) 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f8388b7e640 1 -- 172.21.15.156:0/2268512854 learned_addr learned my addr 172.21.15.156:0/2268512854 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f837bfff640 1 --2- 172.21.15.156:0/2268512854 >> v2:172.21.15.203:3300/0 conn(0x7f83840d2120 0x7f838406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838937f640 1 -- 172.21.15.156:0/2268512854 >> v2:172.21.15.203:3300/0 conn(0x7f83840d2120 msgr2=0x7f838406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838937f640 1 --2- 172.21.15.156:0/2268512854 >> v2:172.21.15.203:3300/0 conn(0x7f83840d2120 0x7f838406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838937f640 1 -- 172.21.15.156:0/2268512854 >> v1:172.21.15.156:6789/0 conn(0x7f838415e920 legacy=0x7f83840d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.032+0000 7f838937f640 1 -- 172.21.15.156:0/2268512854 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83840d3080 con 0x7f838406cb10 2024-03-29T04:09:25.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.033+0000 7f838937f640 1 --2- 172.21.15.156:0/2268512854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838406cb10 0x7f83840cfc30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f838408c830 tx=0x7f837c02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c0ce5933553367 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.033+0000 7f837b7fe640 1 -- 172.21.15.156:0/2268512854 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c039070 con 0x7f838406cb10 2024-03-29T04:09:25.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.033+0000 7f837b7fe640 1 -- 172.21.15.156:0/2268512854 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f837c02ca30 con 0x7f838406cb10 2024-03-29T04:09:25.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.033+0000 7f837b7fe640 1 -- 172.21.15.156:0/2268512854 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c02ce00 con 0x7f838406cb10 2024-03-29T04:09:25.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 -- 172.21.15.156:0/2268512854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838406cb10 msgr2=0x7f83840cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 --2- 172.21.15.156:0/2268512854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838406cb10 0x7f83840cfc30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f838408c830 tx=0x7f837c02bd00 comp rx=0 tx=0).stop 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 -- 172.21.15.156:0/2268512854 shutdown_connections 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 --2- 172.21.15.156:0/2268512854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838406cb10 0x7f83840cfc30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 --2- 172.21.15.156:0/2268512854 >> v2:172.21.15.203:3300/0 conn(0x7f83840d2120 0x7f838406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 -- 172.21.15.156:0/2268512854 >> 172.21.15.156:0/2268512854 conn(0x7f838409d7a0 msgr2=0x7f83840d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 -- 172.21.15.156:0/2268512854 shutdown_connections 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.034+0000 7f838ade2640 1 -- 172.21.15.156:0/2268512854 wait complete. 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.035+0000 7f838ade2640 1 Processor -- start 2024-03-29T04:09:25.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.035+0000 7f838ade2640 1 -- start start 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f838406cb10 0x7f838415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83840d2120 0x7f83840b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 0x7f83840b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83840d34e0 con 0x7f838415e920 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83840d3360 con 0x7f838406cb10 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f83840d0170 con 0x7f83840d2120 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 0x7f83840b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 0x7f83840b27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42686/0 (socket says 172.21.15.156:42686) 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 -- 172.21.15.156:0/846519059 learned_addr learned my addr 172.21.15.156:0/846519059 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 -- 172.21.15.156:0/846519059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83840d2120 msgr2=0x7f83840b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83840d2120 0x7f83840b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 -- 172.21.15.156:0/846519059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f838406cb10 msgr2=0x7f838415fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f838406cb10 0x7f838415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 -- 172.21.15.156:0/846519059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83840ccb70 con 0x7f838415e920 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838937f640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 0x7f83840b27c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f837c009e80 tx=0x7f837c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c039050 con 0x7f838415e920 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f837c045070 con 0x7f838415e920 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c040450 con 0x7f838415e920 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.036+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83841f53d0 con 0x7f838415e920 2024-03-29T04:09:25.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.037+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83841f4b30 con 0x7f838415e920 2024-03-29T04:09:25.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.038+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83840a2480 con 0x7f838415e920 2024-03-29T04:09:25.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.038+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f837c0405f0 con 0x7f838415e920 2024-03-29T04:09:25.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.038+0000 7f83797fa640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f837006c530 0x7f837006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.038+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f837c0adfa0 con 0x7f838415e920 2024-03-29T04:09:25.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.038+0000 7f8388b7e640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f837006c530 0x7f837006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.039+0000 7f8388b7e640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f837006c530 0x7f837006e9f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8380007fd0 tx=0x7f83800079a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.022 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.043+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f83840ccb70 con 0x7f838415e920 2024-03-29T04:09:25.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.285+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f838404f9d0 con 0x7f838415e920 2024-03-29T04:09:25.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.288+0000 7f83797fa640 1 -- 172.21.15.156:0/846519059 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f837c07c650 con 0x7f838415e920 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f837006c530 msgr2=0x7f837006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f837006c530 0x7f837006e9f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8380007fd0 tx=0x7f83800079a0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 msgr2=0x7f83840b27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 0x7f83840b27c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f837c009e80 tx=0x7f837c004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 shutdown_connections 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f837006c530 0x7f837006e9f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838415e920 0x7f83840b27c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f83840d2120 0x7f83840b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 --2- 172.21.15.156:0/846519059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f838406cb10 0x7f838415fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 >> 172.21.15.156:0/846519059 conn(0x7f838409d7a0 msgr2=0x7f83840b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:25.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 shutdown_connections 2024-03-29T04:09:25.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.290+0000 7f838ade2640 1 -- 172.21.15.156:0/846519059 wait complete. 2024-03-29T04:09:25.280 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T04:09:25.281 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T04:09:25.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T04:09:25.384 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T04:09:25.384 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:09:25.384 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:25.400 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:09:25.401 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:25.455 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:25.611 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:09:25.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:09:25.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 Processor -- start 2024-03-29T04:09:25.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 -- start start 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbb90070700 0x7fbb9006b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbb9006baf0 0x7fbb90158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb900b44a0 con 0x7fbb90070320 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbb9006bf60 con 0x7fbb9006baf0 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.650+0000 7fbb963db640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbb900b4780 con 0x7fbb90070700 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb94978640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbb9006baf0 0x7fbb90158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb8ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbb90070320 0x7fbb9015f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57996/0 (socket says 172.21.15.156:57996) 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb8ffff640 1 -- 172.21.15.156:0/847269434 learned_addr learned my addr 172.21.15.156:0/847269434 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb8f7fe640 1 --2- 172.21.15.156:0/847269434 >> v2:172.21.15.203:3300/0 conn(0x7fbb90070700 0x7fbb9006b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb94978640 1 -- 172.21.15.156:0/847269434 >> v2:172.21.15.203:3300/0 conn(0x7fbb90070700 msgr2=0x7fbb9006b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb94978640 1 --2- 172.21.15.156:0/847269434 >> v2:172.21.15.203:3300/0 conn(0x7fbb90070700 0x7fbb9006b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb94978640 1 -- 172.21.15.156:0/847269434 >> v1:172.21.15.156:6789/0 conn(0x7fbb90070320 legacy=0x7fbb9015f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb94978640 1 -- 172.21.15.156:0/847269434 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900b6170 con 0x7fbb9006baf0 2024-03-29T04:09:25.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.651+0000 7fbb8effd640 1 -- 172.21.15.156:0/847269434 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141842067 0 0) 0x7fbb900b44a0 con 0x7fbb90070320 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.652+0000 7fbb94978640 1 --2- 172.21.15.156:0/847269434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb9006baf0 0x7fbb90158460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbb900b3b30 tx=0x7fbb8802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b61e905648d869b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.652+0000 7fbb8effd640 1 -- 172.21.15.156:0/847269434 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88039070 con 0x7fbb9006baf0 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.652+0000 7fbb8effd640 1 -- 172.21.15.156:0/847269434 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8802cc80 con 0x7fbb9006baf0 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.652+0000 7fbb8effd640 1 -- 172.21.15.156:0/847269434 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88034660 con 0x7fbb9006baf0 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.652+0000 7fbb963db640 1 -- 172.21.15.156:0/847269434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb9006baf0 msgr2=0x7fbb90158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 --2- 172.21.15.156:0/847269434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb9006baf0 0x7fbb90158460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbb900b3b30 tx=0x7fbb8802bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 -- 172.21.15.156:0/847269434 shutdown_connections 2024-03-29T04:09:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 --2- 172.21.15.156:0/847269434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb9006baf0 0x7fbb90158460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 --2- 172.21.15.156:0/847269434 >> v2:172.21.15.203:3300/0 conn(0x7fbb90070700 0x7fbb9006b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 -- 172.21.15.156:0/847269434 >> 172.21.15.156:0/847269434 conn(0x7fbb900b68b0 msgr2=0x7fbb900b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:25.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 -- 172.21.15.156:0/847269434 shutdown_connections 2024-03-29T04:09:25.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.653+0000 7fbb963db640 1 -- 172.21.15.156:0/847269434 wait complete. 2024-03-29T04:09:25.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.654+0000 7fbb963db640 1 Processor -- start 2024-03-29T04:09:25.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.654+0000 7fbb963db640 1 -- start start 2024-03-29T04:09:25.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9006baf0 0x7fbb90157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb90070320 0x7fbb90159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 0x7fbb90159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90072f10 con 0x7fbb90070700 2024-03-29T04:09:25.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb900b4170 con 0x7fbb9006baf0 2024-03-29T04:09:25.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbb90073090 con 0x7fbb90070320 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 0x7fbb90159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 0x7fbb90159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42710/0 (socket says 172.21.15.156:42710) 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 -- 172.21.15.156:0/2474680270 learned_addr learned my addr 172.21.15.156:0/2474680270 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb8ffff640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9006baf0 0x7fbb90157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb8f7fe640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb90070320 0x7fbb90159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 -- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb90070320 msgr2=0x7fbb90159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb90070320 0x7fbb90159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 -- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9006baf0 msgr2=0x7fbb90157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9006baf0 0x7fbb90157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 -- 172.21.15.156:0/2474680270 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb90157a70 con 0x7fbb90070700 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb94978640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 0x7fbb90159c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb88037040 tx=0x7fbb88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88039050 con 0x7fbb90070700 2024-03-29T04:09:25.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb88045070 con 0x7fbb90070700 2024-03-29T04:09:25.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb900cb2a0 con 0x7fbb90070700 2024-03-29T04:09:25.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88009450 con 0x7fbb90070700 2024-03-29T04:09:25.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.655+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb900cb860 con 0x7fbb90070700 2024-03-29T04:09:25.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.656+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb90068d20 con 0x7fbb90070700 2024-03-29T04:09:25.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.657+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbb8803e850 con 0x7fbb90070700 2024-03-29T04:09:25.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.657+0000 7fbb8cff9640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbb7c06c530 0x7fbb7c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.658+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb880ae5d0 con 0x7fbb90070700 2024-03-29T04:09:25.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.658+0000 7fbb8ffff640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbb7c06c530 0x7fbb7c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.658+0000 7fbb8ffff640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbb7c06c530 0x7fbb7c06e9f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb80000f60 tx=0x7fbb80007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.664+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbb8807cc80 con 0x7fbb90070700 2024-03-29T04:09:25.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.875+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb9006f940 con 0x7fbb90070700 2024-03-29T04:09:25.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.876+0000 7fbb8cff9640 1 -- 172.21.15.156:0/2474680270 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fbb8807c580 con 0x7fbb90070700 2024-03-29T04:09:25.856 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:25.856 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:25.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbb7c06c530 msgr2=0x7fbb7c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbb7c06c530 0x7fbb7c06e9f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb80000f60 tx=0x7fbb80007d50 comp rx=0 tx=0).stop 2024-03-29T04:09:25.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 msgr2=0x7fbb90159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 0x7fbb90159c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb88037040 tx=0x7fbb88004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:25.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 shutdown_connections 2024-03-29T04:09:25.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbb7c06c530 0x7fbb7c06e9f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb90070700 0x7fbb90159c80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb90070320 0x7fbb90159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 --2- 172.21.15.156:0/2474680270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9006baf0 0x7fbb90157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.878+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 >> 172.21.15.156:0/2474680270 conn(0x7fbb900b68b0 msgr2=0x7fbb9006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:25.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.879+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 shutdown_connections 2024-03-29T04:09:25.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.879+0000 7fbb963db640 1 -- 172.21.15.156:0/2474680270 wait complete. 2024-03-29T04:09:25.869 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:25.870 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:25.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.996+0000 7fa938b24640 1 Processor -- start 2024-03-29T04:09:25.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.996+0000 7fa938b24640 1 -- start start 2024-03-29T04:09:25.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa938b24640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa93415e920 0x7fa9340d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa938b24640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9340cfc30 0x7fa9340d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa938b24640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9340cf190 con 0x7fa9340d2120 2024-03-29T04:09:25.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa938b24640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa9340cf7a0 con 0x7fa93415e920 2024-03-29T04:09:25.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa938b24640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa9340d3080 con 0x7fa9340cfc30 2024-03-29T04:09:25.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa93259c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa93415e920 0x7fa9340d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa931d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa9340d2120 0x7fa93406c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57998/0 (socket says 172.21.15.156:57998) 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa931d9b640 1 -- 172.21.15.156:0/4230918337 learned_addr learned my addr 172.21.15.156:0/4230918337 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa932d9d640 1 --2- 172.21.15.156:0/4230918337 >> v2:172.21.15.203:3300/0 conn(0x7fa9340cfc30 0x7fa9340d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa93259c640 1 -- 172.21.15.156:0/4230918337 >> v2:172.21.15.203:3300/0 conn(0x7fa9340cfc30 msgr2=0x7fa9340d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa93259c640 1 --2- 172.21.15.156:0/4230918337 >> v2:172.21.15.203:3300/0 conn(0x7fa9340cfc30 0x7fa9340d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa93259c640 1 -- 172.21.15.156:0/4230918337 >> v1:172.21.15.156:6789/0 conn(0x7fa9340d2120 legacy=0x7fa93406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa93259c640 1 -- 172.21.15.156:0/4230918337 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9340cf430 con 0x7fa93415e920 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.998+0000 7fa93159a640 1 -- 172.21.15.156:0/4230918337 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2083745037 0 0) 0x7fa9340cf190 con 0x7fa9340d2120 2024-03-29T04:09:25.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.999+0000 7fa932d9d640 1 --2- 172.21.15.156:0/4230918337 >> v2:172.21.15.203:3300/0 conn(0x7fa9340cfc30 0x7fa9340d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:25.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.999+0000 7fa93259c640 1 --2- 172.21.15.156:0/4230918337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa93415e920 0x7fa9340d26e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa93408ca00 tx=0x7fa92c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5406c33c3fddc1c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.999+0000 7fa93159a640 1 -- 172.21.15.156:0/4230918337 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c035020 con 0x7fa93415e920 2024-03-29T04:09:25.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.999+0000 7fa93159a640 1 -- 172.21.15.156:0/4230918337 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92c031830 con 0x7fa93415e920 2024-03-29T04:09:25.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:25.999+0000 7fa93159a640 1 -- 172.21.15.156:0/4230918337 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c031c00 con 0x7fa93415e920 2024-03-29T04:09:25.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 -- 172.21.15.156:0/4230918337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa93415e920 msgr2=0x7fa9340d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 --2- 172.21.15.156:0/4230918337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa93415e920 0x7fa9340d26e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa93408ca00 tx=0x7fa92c02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 -- 172.21.15.156:0/4230918337 shutdown_connections 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 --2- 172.21.15.156:0/4230918337 >> v2:172.21.15.203:3300/0 conn(0x7fa9340cfc30 0x7fa9340d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 --2- 172.21.15.156:0/4230918337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa93415e920 0x7fa9340d26e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 -- 172.21.15.156:0/4230918337 >> 172.21.15.156:0/4230918337 conn(0x7fa93409d7a0 msgr2=0x7fa9340d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 -- 172.21.15.156:0/4230918337 shutdown_connections 2024-03-29T04:09:25.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.000+0000 7fa938b24640 1 -- 172.21.15.156:0/4230918337 wait complete. 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.001+0000 7fa938b24640 1 Processor -- start 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.001+0000 7fa938b24640 1 -- start start 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa938b24640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9340cfc30 0x7fa93409e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa938b24640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 0x7fa9340ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa938b24640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93415e920 0x7fa93415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa938b24640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340d34e0 con 0x7fa9340d2120 2024-03-29T04:09:25.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa938b24640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340d3360 con 0x7fa9340cfc30 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa938b24640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa9340d2c20 con 0x7fa93415e920 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa931d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 0x7fa9340ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa931d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 0x7fa9340ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42742/0 (socket says 172.21.15.156:42742) 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa931d9b640 1 -- 172.21.15.156:0/2986309059 learned_addr learned my addr 172.21.15.156:0/2986309059 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa93259c640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9340cfc30 0x7fa93409e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.002+0000 7fa932d9d640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93415e920 0x7fa93415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa931d9b640 1 -- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93415e920 msgr2=0x7fa93415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa931d9b640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93415e920 0x7fa93415fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa931d9b640 1 -- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9340cfc30 msgr2=0x7fa93409e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa931d9b640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9340cfc30 0x7fa93409e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa931d9b640 1 -- 172.21.15.156:0/2986309059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9340ce3d0 con 0x7fa9340d2120 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa931d9b640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 0x7fa9340ca380 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa92800c500 tx=0x7fa92800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928018070 con 0x7fa9340d2120 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa928007580 con 0x7fa9340d2120 2024-03-29T04:09:25.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928004a20 con 0x7fa9340d2120 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9340ca8c0 con 0x7fa9340d2120 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.003+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9340ce640 con 0x7fa9340d2120 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.004+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9340a2480 con 0x7fa9340d2120 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.005+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa92801b070 con 0x7fa9340d2120 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.005+0000 7fa922ffd640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa91406c530 0x7fa91406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.005+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa92808a900 con 0x7fa9340d2120 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.005+0000 7fa93259c640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa91406c530 0x7fa91406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.006+0000 7fa93259c640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa91406c530 0x7fa91406e9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa92c036000 tx=0x7fa92c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:25.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.010+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa928058960 con 0x7fa9340d2120 2024-03-29T04:09:26.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.186+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa93406c950 con 0x7fa9340d2120 2024-03-29T04:09:26.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.186+0000 7fa922ffd640 1 -- 172.21.15.156:0/2986309059 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fa92805cc60 con 0x7fa9340d2120 2024-03-29T04:09:26.166 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa91406c530 msgr2=0x7fa91406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa91406c530 0x7fa91406e9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa92c036000 tx=0x7fa92c03c040 comp rx=0 tx=0).stop 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 msgr2=0x7fa9340ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 0x7fa9340ca380 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa92800c500 tx=0x7fa92800c9c0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 shutdown_connections 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa91406c530 0x7fa91406e9f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93415e920 0x7fa93415fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9340d2120 0x7fa9340ca380 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 --2- 172.21.15.156:0/2986309059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9340cfc30 0x7fa93409e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 >> 172.21.15.156:0/2986309059 conn(0x7fa93409d7a0 msgr2=0x7fa93406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 shutdown_connections 2024-03-29T04:09:26.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.188+0000 7fa938b24640 1 -- 172.21.15.156:0/2986309059 wait complete. 2024-03-29T04:09:26.179 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:26.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.304+0000 7f7ec998f640 1 Processor -- start 2024-03-29T04:09:26.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.304+0000 7f7ec998f640 1 -- start start 2024-03-29T04:09:26.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec998f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7ec415e920 0x7f7ec40d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec998f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 0x7f7ec406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec998f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec40cf190 con 0x7f7ec406cb10 2024-03-29T04:09:26.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec998f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7ec40cf7a0 con 0x7f7ec415e920 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec998f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec40d3080 con 0x7f7ec40d2120 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec37fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7ec406cb10 0x7f7ec40cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58012/0 (socket says 172.21.15.156:58012) 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec37fe640 1 -- 172.21.15.156:0/1446558755 learned_addr learned my addr 172.21.15.156:0/1446558755 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 683451596 0 0) 0x7f7ec40cf190 con 0x7f7ec406cb10 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb0003520 con 0x7f7ec406cb10 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec2ffd640 1 --2- 172.21.15.156:0/1446558755 >> v2:172.21.15.203:3300/0 conn(0x7f7ec415e920 0x7f7ec40d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7eba7fc640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 0x7f7ec406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1593258332 0 0) 0x7f7eb0003520 con 0x7f7ec406cb10 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 msgr2=0x7f7ec406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 0x7f7ec406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 >> v2:172.21.15.203:3300/0 conn(0x7f7ec415e920 msgr2=0x7f7ec40d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 --2- 172.21.15.156:0/1446558755 >> v2:172.21.15.203:3300/0 conn(0x7f7ec415e920 0x7f7ec40d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec40cf430 con 0x7f7ec406cb10 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f7ebc0019f0 con 0x7f7ec406cb10 2024-03-29T04:09:26.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.305+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 >> v1:172.21.15.156:6789/0 conn(0x7f7ec406cb10 legacy=0x7f7ec40cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec27fc640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7eb00039b0 0x7f7eb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec27fc640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 0x7f7eb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec27fc640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7eb0004d40 0x7f7eb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec40cf7a0 con 0x7f7eb0004d40 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec40d3080 con 0x7f7eb00039b0 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb000b9c0 con 0x7f7ec40d2120 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec37fe640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7eb0004d40 0x7f7eb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec2ffd640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7eb00039b0 0x7f7eb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7eba7fc640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 0x7f7eb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec37fe640 1 -- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 msgr2=0x7f7eb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec37fe640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40d2120 0x7f7eb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec37fe640 1 -- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7eb00039b0 msgr2=0x7f7eb0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec37fe640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7eb00039b0 0x7f7eb0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.306+0000 7f7ec37fe640 1 -- 172.21.15.156:0/1446558755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb0003520 con 0x7f7eb0004d40 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec37fe640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7eb0004d40 0x7f7eb00051c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0013d0 tx=0x7f7eb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e6c51e75955886 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc00a410 con 0x7f7eb0004d40 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc034630 con 0x7f7eb0004d40 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec27fc640 1 -- 172.21.15.156:0/1446558755 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc034a20 con 0x7f7eb0004d40 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec998f640 1 -- 172.21.15.156:0/1446558755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7eb0004d40 msgr2=0x7f7eb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec998f640 1 --2- 172.21.15.156:0/1446558755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7eb0004d40 0x7f7eb00051c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0013d0 tx=0x7f7eb0005830 comp rx=0 tx=0).stop 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec37fe640 1 -- 172.21.15.156:0/1446558755 reap_dead start 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec998f640 1 -- 172.21.15.156:0/1446558755 shutdown_connections 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec998f640 1 -- 172.21.15.156:0/1446558755 >> 172.21.15.156:0/1446558755 conn(0x7f7ec409d7a0 msgr2=0x7f7ec40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:26.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec998f640 1 -- 172.21.15.156:0/1446558755 shutdown_connections 2024-03-29T04:09:26.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.307+0000 7f7ec998f640 1 -- 172.21.15.156:0/1446558755 wait complete. 2024-03-29T04:09:26.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 Processor -- start 2024-03-29T04:09:26.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 -- start start 2024-03-29T04:09:26.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40ca2f0 0x7f7ec40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 0x7f7ec41f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec41f8380 0x7f7ec40cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec40d04c0 con 0x7f7ec409e210 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec40d0340 con 0x7f7ec41f8380 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.309+0000 7f7ec998f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec40d2c20 con 0x7f7ec40ca2f0 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 0x7f7ec41f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 0x7f7ec41f7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42752/0 (socket says 172.21.15.156:42752) 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 -- 172.21.15.156:0/2985465725 learned_addr learned my addr 172.21.15.156:0/2985465725 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec37fe640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec41f8380 0x7f7ec40cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec2ffd640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40ca2f0 0x7f7ec40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 -- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40ca2f0 msgr2=0x7f7ec40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40ca2f0 0x7f7ec40ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 -- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec41f8380 msgr2=0x7f7ec40cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec41f8380 0x7f7ec40cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 -- 172.21.15.156:0/2985465725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec415bf40 con 0x7f7ec409e210 2024-03-29T04:09:26.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec37fe640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec41f8380 0x7f7ec40cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7eba7fc640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 0x7f7ec41f7e40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7eac011a00 tx=0x7f7eac011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac00fa90 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eac00fc30 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec40cbd00 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac014440 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.310+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec409e780 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.311+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec40a2480 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.312+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7eac012040 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.312+0000 7f7ec0ff9640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ea406c530 0x7f7ea406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.312+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7eac08bbe0 con 0x7f7ec409e210 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.312+0000 7f7ec2ffd640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ea406c530 0x7f7ea406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.313+0000 7f7ec2ffd640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ea406c530 0x7f7ea406e9f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009c40 tx=0x7f7eb4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.317+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7eac019030 con 0x7f7ec409e210 2024-03-29T04:09:26.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.498+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7ec404f9d0 con 0x7f7ec409e210 2024-03-29T04:09:26.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.498+0000 7f7ec0ff9640 1 -- 172.21.15.156:0/2985465725 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f7eac05a290 con 0x7f7ec409e210 2024-03-29T04:09:26.478 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:26.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ea406c530 msgr2=0x7f7ea406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ea406c530 0x7f7ea406e9f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009c40 tx=0x7f7eb4009190 comp rx=0 tx=0).stop 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 msgr2=0x7f7ec41f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 0x7f7ec41f7e40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7eac011a00 tx=0x7f7eac011e00 comp rx=0 tx=0).stop 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 shutdown_connections 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ea406c530 0x7f7ea406e9f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec41f8380 0x7f7ec40cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec409e210 0x7f7ec41f7e40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 --2- 172.21.15.156:0/2985465725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ec40ca2f0 0x7f7ec40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.500+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 >> 172.21.15.156:0/2985465725 conn(0x7f7ec409d7a0 msgr2=0x7f7ec415d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.501+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 shutdown_connections 2024-03-29T04:09:26.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.501+0000 7f7ec998f640 1 -- 172.21.15.156:0/2985465725 wait complete. 2024-03-29T04:09:26.491 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:26.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.616+0000 7fee65182640 1 Processor -- start 2024-03-29T04:09:26.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.616+0000 7fee65182640 1 -- start start 2024-03-29T04:09:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee65182640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fee600d2120 0x7fee6006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee65182640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fee6006cb10 0x7fee600cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee65182640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee600d2df0 con 0x7fee6015e920 2024-03-29T04:09:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee65182640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fee600cf7a0 con 0x7fee600d2120 2024-03-29T04:09:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee65182640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fee600cf190 con 0x7fee6006cb10 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee5e59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fee600d2120 0x7fee6006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee5e59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fee600d2120 0x7fee6006c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42760/0 (socket says 172.21.15.156:42760) 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.617+0000 7fee5e59c640 1 -- 172.21.15.156:0/407706400 learned_addr learned my addr 172.21.15.156:0/407706400 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.618+0000 7fee5f59e640 1 --2- 172.21.15.156:0/407706400 >> v2:172.21.15.203:3300/0 conn(0x7fee6006cb10 0x7fee600cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.618+0000 7fee5e59c640 1 -- 172.21.15.156:0/407706400 >> v2:172.21.15.203:3300/0 conn(0x7fee6006cb10 msgr2=0x7fee600cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.618+0000 7fee5e59c640 1 --2- 172.21.15.156:0/407706400 >> v2:172.21.15.203:3300/0 conn(0x7fee6006cb10 0x7fee600cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.618+0000 7fee5e59c640 1 -- 172.21.15.156:0/407706400 >> v1:172.21.15.203:6789/0 conn(0x7fee6015e920 legacy=0x7fee600d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.618+0000 7fee5e59c640 1 -- 172.21.15.156:0/407706400 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee600d3080 con 0x7fee600d2120 2024-03-29T04:09:26.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.619+0000 7fee5e59c640 1 --2- 172.21.15.156:0/407706400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee600d2120 0x7fee6006c5d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fee6008ca00 tx=0x7fee5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a53841398e2a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.619+0000 7fee5dd9b640 1 -- 172.21.15.156:0/407706400 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54034070 con 0x7fee600d2120 2024-03-29T04:09:26.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.619+0000 7fee5dd9b640 1 -- 172.21.15.156:0/407706400 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee54030860 con 0x7fee600d2120 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.619+0000 7fee5dd9b640 1 -- 172.21.15.156:0/407706400 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54030c30 con 0x7fee600d2120 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.619+0000 7fee65182640 1 -- 172.21.15.156:0/407706400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee600d2120 msgr2=0x7fee6006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 --2- 172.21.15.156:0/407706400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee600d2120 0x7fee6006c5d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fee6008ca00 tx=0x7fee5402b9a0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 -- 172.21.15.156:0/407706400 shutdown_connections 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 --2- 172.21.15.156:0/407706400 >> v2:172.21.15.203:3300/0 conn(0x7fee6006cb10 0x7fee600cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 --2- 172.21.15.156:0/407706400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee600d2120 0x7fee6006c5d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 -- 172.21.15.156:0/407706400 >> 172.21.15.156:0/407706400 conn(0x7fee6009d7a0 msgr2=0x7fee600d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 -- 172.21.15.156:0/407706400 shutdown_connections 2024-03-29T04:09:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.620+0000 7fee65182640 1 -- 172.21.15.156:0/407706400 wait complete. 2024-03-29T04:09:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.621+0000 7fee65182640 1 Processor -- start 2024-03-29T04:09:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.621+0000 7fee65182640 1 -- start start 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee65182640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee6006cb10 0x7fee600ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee65182640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee600d2120 0x7fee601f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee65182640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 0x7fee600c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee65182640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fee600d34e0 con 0x7fee6015e920 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee65182640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee600d3360 con 0x7fee6006cb10 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee65182640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fee600d0170 con 0x7fee600d2120 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 0x7fee600c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 0x7fee600c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42764/0 (socket says 172.21.15.156:42764) 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 -- 172.21.15.156:0/4101925496 learned_addr learned my addr 172.21.15.156:0/4101925496 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5ed9d640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee6006cb10 0x7fee600ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5e59c640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee600d2120 0x7fee601f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 -- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee600d2120 msgr2=0x7fee601f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee600d2120 0x7fee601f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 -- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee6006cb10 msgr2=0x7fee600ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee6006cb10 0x7fee600ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.622+0000 7fee5f59e640 1 -- 172.21.15.156:0/4101925496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee601f48c0 con 0x7fee6015e920 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.623+0000 7fee5f59e640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 0x7fee600c7d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee5800d800 tx=0x7fee5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.623+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5800b6c0 con 0x7fee6015e920 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.623+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee58019070 con 0x7fee6015e920 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.623+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee600cd850 con 0x7fee6015e920 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.623+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5800bac0 con 0x7fee6015e920 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.623+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee600b4150 con 0x7fee6015e920 2024-03-29T04:09:26.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.624+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee600a2480 con 0x7fee6015e920 2024-03-29T04:09:26.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.624+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fee5800bc60 con 0x7fee6015e920 2024-03-29T04:09:26.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.625+0000 7fee477fe640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fee4c06c530 0x7fee4c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.625+0000 7fee5ed9d640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fee4c06c530 0x7fee4c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.625+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fee5808baa0 con 0x7fee6015e920 2024-03-29T04:09:26.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.625+0000 7fee5ed9d640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fee4c06c530 0x7fee4c06e9f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fee50001ca0 tx=0x7fee50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.629+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fee5805a150 con 0x7fee6015e920 2024-03-29T04:09:26.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.806+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee600d1d80 con 0x7fee6015e920 2024-03-29T04:09:26.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.806+0000 7fee477fe640 1 -- 172.21.15.156:0/4101925496 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fee58059a50 con 0x7fee6015e920 2024-03-29T04:09:26.786 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:26.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fee4c06c530 msgr2=0x7fee4c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fee4c06c530 0x7fee4c06e9f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fee50001ca0 tx=0x7fee50009040 comp rx=0 tx=0).stop 2024-03-29T04:09:26.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 msgr2=0x7fee600c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 0x7fee600c7d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee5800d800 tx=0x7fee5800dcc0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 shutdown_connections 2024-03-29T04:09:26.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fee4c06c530 0x7fee4c06e9f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee6015e920 0x7fee600c7d80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee600d2120 0x7fee601f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 --2- 172.21.15.156:0/4101925496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee6006cb10 0x7fee600ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 >> 172.21.15.156:0/4101925496 conn(0x7fee6009d7a0 msgr2=0x7fee6015d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:26.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 shutdown_connections 2024-03-29T04:09:26.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.808+0000 7fee65182640 1 -- 172.21.15.156:0/4101925496 wait complete. 2024-03-29T04:09:26.801 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:26.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.924+0000 7f2d38214640 1 Processor -- start 2024-03-29T04:09:26.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.924+0000 7f2d38214640 1 -- start start 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.925+0000 7f2d38214640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2d30145c70 0x7f2d300b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d38214640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3015cb10 con 0x7f2d300cf510 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d38214640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3015cdf0 con 0x7f2d300b4310 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d38214640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2d300cf980 con 0x7f2d30145c70 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d367b1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2d300cf510 0x7f2d300d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58024/0 (socket says 172.21.15.156:58024) 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d367b1640 1 -- 172.21.15.156:0/834532410 learned_addr learned my addr 172.21.15.156:0/834532410 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d35fb0640 1 --2- 172.21.15.156:0/834532410 >> v2:172.21.15.203:3300/0 conn(0x7f2d30145c70 0x7f2d300b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 376219695 0 0) 0x7f2d3015cb10 con 0x7f2d300cf510 2024-03-29T04:09:26.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d1c003520 con 0x7f2d300cf510 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3866442704 0 0) 0x7f2d1c003520 con 0x7f2d300cf510 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 >> v2:172.21.15.203:3300/0 conn(0x7f2d30145c70 msgr2=0x7f2d300b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 --2- 172.21.15.156:0/834532410 >> v2:172.21.15.203:3300/0 conn(0x7f2d30145c70 0x7f2d300b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 >> v1:172.21.15.203:6789/0 conn(0x7f2d300b4310 legacy=0x7f2d300cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3015d0d0 con 0x7f2d300cf510 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f2d280019f0 con 0x7f2d300cf510 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 >> v1:172.21.15.156:6789/0 conn(0x7f2d300cf510 legacy=0x7f2d300d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d1c0039b0 0x7f2d1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d300cf980 con 0x7f2d1c004800 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c00c160 con 0x7f2d1c0051c0 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.926+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1c00c360 con 0x7f2d1c0039b0 2024-03-29T04:09:26.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d357af640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d35fb0640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d1c0039b0 0x7f2d1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d367b1640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d357af640 1 -- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d1c0039b0 msgr2=0x7f2d1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d357af640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d1c0039b0 0x7f2d1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d357af640 1 -- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d1c0051c0 msgr2=0x7f2d1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d357af640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d367b1640 1 -- 172.21.15.156:0/834532410 reap_dead start 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.927+0000 7f2d357af640 1 -- 172.21.15.156:0/834532410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c003520 con 0x7f2d1c004800 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d357af640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d1c004c80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c001750 tx=0x7f2d1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b016babe8bb1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c001370 con 0x7f2d1c004800 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c0052d0 con 0x7f2d1c004800 2024-03-29T04:09:26.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d34fae640 1 -- 172.21.15.156:0/834532410 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c003830 con 0x7f2d1c004800 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 -- 172.21.15.156:0/834532410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 msgr2=0x7f2d1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d1c004c80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c001750 tx=0x7f2d1c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 -- 172.21.15.156:0/834532410 shutdown_connections 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 --2- 172.21.15.156:0/834532410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d1c004c80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 -- 172.21.15.156:0/834532410 >> 172.21.15.156:0/834532410 conn(0x7f2d300b7660 msgr2=0x7f2d300b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 -- 172.21.15.156:0/834532410 shutdown_connections 2024-03-29T04:09:26.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.928+0000 7f2d38214640 1 -- 172.21.15.156:0/834532410 wait complete. 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.929+0000 7f2d38214640 1 Processor -- start 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.929+0000 7f2d38214640 1 -- start start 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d38214640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d301f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d38214640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30140dd0 0x7f2d301f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d38214640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d30141ba0 0x7f2d30159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d38214640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301461d0 con 0x7f2d1c004800 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d38214640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d30146050 con 0x7f2d30140dd0 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d38214640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2d300b3dd0 con 0x7f2d30141ba0 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d35fb0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d301f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d35fb0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d301f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42776/0 (socket says 172.21.15.156:42776) 2024-03-29T04:09:26.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d35fb0640 1 -- 172.21.15.156:0/14206972 learned_addr learned my addr 172.21.15.156:0/14206972 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.930+0000 7f2d357af640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30140dd0 0x7f2d301f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d367b1640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d30141ba0 0x7f2d30159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d35fb0640 1 -- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d30141ba0 msgr2=0x7f2d30159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d35fb0640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d30141ba0 0x7f2d30159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d35fb0640 1 -- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30140dd0 msgr2=0x7f2d301f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d35fb0640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30140dd0 0x7f2d301f6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d35fb0640 1 -- 172.21.15.156:0/14206972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d30141250 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d35fb0640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d301f2a00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2d20011a70 tx=0x7f2d20011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2000fb50 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2000fcf0 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d301f6f70 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2000fea0 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.931+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d301f3340 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.932+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d30070e10 con 0x7f2d1c004800 2024-03-29T04:09:26.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.932+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2d20012040 con 0x7f2d1c004800 2024-03-29T04:09:26.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.933+0000 7f2d267fc640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d1406c560 0x7f2d1406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:26.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.933+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2d2008be30 con 0x7f2d1c004800 2024-03-29T04:09:26.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.933+0000 7f2d357af640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d1406c560 0x7f2d1406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:26.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.933+0000 7f2d357af640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d1406c560 0x7f2d1406ea20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c033000 tx=0x7f2d2c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:26.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:26.941+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2d301f6f70 con 0x7f2d1c004800 2024-03-29T04:09:27.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.244+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2d300ce300 con 0x7f2d1c004800 2024-03-29T04:09:27.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.244+0000 7f2d267fc640 1 -- 172.21.15.156:0/14206972 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f2d20019030 con 0x7f2d1c004800 2024-03-29T04:09:27.224 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d1406c560 msgr2=0x7f2d1406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d1406c560 0x7f2d1406ea20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c033000 tx=0x7f2d2c009030 comp rx=0 tx=0).stop 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 msgr2=0x7f2d301f2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d301f2a00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2d20011a70 tx=0x7f2d20011f30 comp rx=0 tx=0).stop 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 shutdown_connections 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d1406c560 0x7f2d1406ea20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d30141ba0 0x7f2d30159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30140dd0 0x7f2d301f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 --2- 172.21.15.156:0/14206972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d1c004800 0x7f2d301f2a00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 >> 172.21.15.156:0/14206972 conn(0x7f2d300b7660 msgr2=0x7f2d30143c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:27.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 shutdown_connections 2024-03-29T04:09:27.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.246+0000 7f2d38214640 1 -- 172.21.15.156:0/14206972 wait complete. 2024-03-29T04:09:27.237 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:09:27.237 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:27.252 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:27.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.414+0000 7f53e8f72640 1 Processor -- start 2024-03-29T04:09:27.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.414+0000 7f53e8f72640 1 -- start start 2024-03-29T04:09:27.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53e8f72640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f53e40d42c0 0x7f53e40d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53e8f72640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e40d4c80 con 0x7f53e415e730 2024-03-29T04:09:27.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53e8f72640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e40cfa20 con 0x7f53e406cbb0 2024-03-29T04:09:27.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53e8f72640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f53e406c720 con 0x7f53e40d42c0 2024-03-29T04:09:27.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53dbfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f53e40d42c0 0x7f53e40d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53e259c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f53e415e730 0x7f53e40ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58036/0 (socket says 172.21.15.156:58036) 2024-03-29T04:09:27.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.415+0000 7f53e259c640 1 -- 172.21.15.156:0/3595174111 learned_addr learned my addr 172.21.15.156:0/3595174111 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:27.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.416+0000 7f53dbfff640 1 -- 172.21.15.156:0/3595174111 >> v1:172.21.15.203:6789/0 conn(0x7f53e406cbb0 legacy=0x7f53e406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:27.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.416+0000 7f53dbfff640 1 -- 172.21.15.156:0/3595174111 >> v1:172.21.15.156:6789/0 conn(0x7f53e415e730 legacy=0x7f53e40ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.416+0000 7f53dbfff640 1 -- 172.21.15.156:0/3595174111 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e40cfd00 con 0x7f53e40d42c0 2024-03-29T04:09:27.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.416+0000 7f53dbfff640 1 --2- 172.21.15.156:0/3595174111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e40d42c0 0x7f53e40d4740 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009790 tx=0x7f53cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec94c9ff0663a585 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:27.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e1d9b640 1 -- 172.21.15.156:0/3595174111 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc035020 con 0x7f53e40d42c0 2024-03-29T04:09:27.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e1d9b640 1 -- 172.21.15.156:0/3595174111 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53cc031830 con 0x7f53e40d42c0 2024-03-29T04:09:27.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e1d9b640 1 -- 172.21.15.156:0/3595174111 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc031c00 con 0x7f53e40d42c0 2024-03-29T04:09:27.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 -- 172.21.15.156:0/3595174111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e40d42c0 msgr2=0x7f53e40d4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 --2- 172.21.15.156:0/3595174111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e40d42c0 0x7f53e40d4740 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009790 tx=0x7f53cc02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 -- 172.21.15.156:0/3595174111 shutdown_connections 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 --2- 172.21.15.156:0/3595174111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e40d42c0 0x7f53e40d4740 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 -- 172.21.15.156:0/3595174111 >> 172.21.15.156:0/3595174111 conn(0x7f53e408bfd0 msgr2=0x7f53e409d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 -- 172.21.15.156:0/3595174111 shutdown_connections 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.417+0000 7f53e8f72640 1 -- 172.21.15.156:0/3595174111 wait complete. 2024-03-29T04:09:27.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.419+0000 7f53e8f72640 1 Processor -- start 2024-03-29T04:09:27.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.419+0000 7f53e8f72640 1 -- start start 2024-03-29T04:09:27.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 0x7f53e415c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53e40d42c0 0x7f53e4148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f53e415e730 0x7f53e40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53e40d2220 con 0x7f53e406cbb0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53e40d02c0 con 0x7f53e40d42c0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f53e415eb10 con 0x7f53e415e730 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 0x7f53e415c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 0x7f53e415c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42796/0 (socket says 172.21.15.156:42796) 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 -- 172.21.15.156:0/2481025062 learned_addr learned my addr 172.21.15.156:0/2481025062 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53dbfff640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53e40d42c0 0x7f53e4148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e2d9d640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f53e415e730 0x7f53e40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 -- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f53e415e730 msgr2=0x7f53e40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f53e415e730 0x7f53e40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 -- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53e40d42c0 msgr2=0x7f53e4148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53e40d42c0 0x7f53e4148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 -- 172.21.15.156:0/2481025062 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e4148cf0 con 0x7f53e406cbb0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e259c640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 0x7f53e415c0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f53d400c3c0 tx=0x7f53d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:27.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d4017070 con 0x7f53e406cbb0 2024-03-29T04:09:27.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d4005ce0 con 0x7f53e406cbb0 2024-03-29T04:09:27.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.421+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e4148fd0 con 0x7f53e406cbb0 2024-03-29T04:09:27.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.422+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d40072f0 con 0x7f53e406cbb0 2024-03-29T04:09:27.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.422+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e4149590 con 0x7f53e406cbb0 2024-03-29T04:09:27.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.423+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e4068610 con 0x7f53e406cbb0 2024-03-29T04:09:27.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.423+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f53d401a070 con 0x7f53e406cbb0 2024-03-29T04:09:27.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.423+0000 7f53daffd640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f53c406c560 0x7f53c406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.423+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f53d408a9a0 con 0x7f53e406cbb0 2024-03-29T04:09:27.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.423+0000 7f53dbfff640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f53c406c560 0x7f53c406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.424+0000 7f53dbfff640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f53c406c560 0x7f53c406ea20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f53cc036000 tx=0x7f53cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:27.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.428+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f53d4058980 con 0x7f53e406cbb0 2024-03-29T04:09:27.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.611+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53e40caea0 con 0x7f53e406cbb0 2024-03-29T04:09:27.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.612+0000 7f53daffd640 1 -- 172.21.15.156:0/2481025062 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f53d405cc80 con 0x7f53e406cbb0 2024-03-29T04:09:27.592 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:27.592 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f53c406c560 msgr2=0x7f53c406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f53c406c560 0x7f53c406ea20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f53cc036000 tx=0x7f53cc008030 comp rx=0 tx=0).stop 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 msgr2=0x7f53e415c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 0x7f53e415c0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f53d400c3c0 tx=0x7f53d400c880 comp rx=0 tx=0).stop 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 shutdown_connections 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f53c406c560 0x7f53c406ea20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f53e415e730 0x7f53e40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53e40d42c0 0x7f53e4148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 --2- 172.21.15.156:0/2481025062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e406cbb0 0x7f53e415c0d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 >> 172.21.15.156:0/2481025062 conn(0x7f53e408bfd0 msgr2=0x7f53e40b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:27.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 shutdown_connections 2024-03-29T04:09:27.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.614+0000 7f53e8f72640 1 -- 172.21.15.156:0/2481025062 wait complete. 2024-03-29T04:09:27.606 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:27.606 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:27.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.731+0000 7f6a42359640 1 Processor -- start 2024-03-29T04:09:27.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.731+0000 7f6a42359640 1 -- start start 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a42359640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6a3c15e920 0x7f6a3c0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a42359640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6a3c0cfc30 0x7f6a3c0d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a42359640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c0cf190 con 0x7f6a3c0d2120 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a42359640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6a3c0cf7a0 con 0x7f6a3c15e920 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a42359640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6a3c0d3080 con 0x7f6a3c0cfc30 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6a3c15e920 0x7f6a3c0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6a3c15e920 0x7f6a3c0d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42810/0 (socket says 172.21.15.156:42810) 2024-03-29T04:09:27.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 -- 172.21.15.156:0/4234733156 learned_addr learned my addr 172.21.15.156:0/4234733156 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:27.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a408f6640 1 --2- 172.21.15.156:0/4234733156 >> v2:172.21.15.203:3300/0 conn(0x7f6a3c0cfc30 0x7f6a3c0d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 -- 172.21.15.156:0/4234733156 >> v2:172.21.15.203:3300/0 conn(0x7f6a3c0cfc30 msgr2=0x7f6a3c0d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:27.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 --2- 172.21.15.156:0/4234733156 >> v2:172.21.15.203:3300/0 conn(0x7f6a3c0cfc30 0x7f6a3c0d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 -- 172.21.15.156:0/4234733156 >> v1:172.21.15.203:6789/0 conn(0x7f6a3c0d2120 legacy=0x7f6a3c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.732+0000 7f6a3bfff640 1 -- 172.21.15.156:0/4234733156 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c0cf430 con 0x7f6a3c15e920 2024-03-29T04:09:27.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.733+0000 7f6a3bfff640 1 --2- 172.21.15.156:0/4234733156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c0d26e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c08ca00 tx=0x7f6a2c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ff463bbabd17c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:27.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.733+0000 7f6a3affd640 1 -- 172.21.15.156:0/4234733156 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c034070 con 0x7f6a3c15e920 2024-03-29T04:09:27.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.733+0000 7f6a3affd640 1 -- 172.21.15.156:0/4234733156 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a2c030860 con 0x7f6a3c15e920 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.733+0000 7f6a3affd640 1 -- 172.21.15.156:0/4234733156 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c030c30 con 0x7f6a3c15e920 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 -- 172.21.15.156:0/4234733156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 msgr2=0x7f6a3c0d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 --2- 172.21.15.156:0/4234733156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c0d26e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c08ca00 tx=0x7f6a2c02b9a0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 -- 172.21.15.156:0/4234733156 shutdown_connections 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 --2- 172.21.15.156:0/4234733156 >> v2:172.21.15.203:3300/0 conn(0x7f6a3c0cfc30 0x7f6a3c0d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 --2- 172.21.15.156:0/4234733156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c0d26e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 -- 172.21.15.156:0/4234733156 >> 172.21.15.156:0/4234733156 conn(0x7f6a3c09d7a0 msgr2=0x7f6a3c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 -- 172.21.15.156:0/4234733156 shutdown_connections 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.734+0000 7f6a42359640 1 -- 172.21.15.156:0/4234733156 wait complete. 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.735+0000 7f6a42359640 1 Processor -- start 2024-03-29T04:09:27.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.735+0000 7f6a42359640 1 -- start start 2024-03-29T04:09:27.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a42359640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a3c0cfc30 0x7f6a3c0c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a42359640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a3c0d2120 0x7f6a3c1e91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a42359640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c1ea060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a42359640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c0d34e0 con 0x7f6a3c15e920 2024-03-29T04:09:27.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a42359640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c0d3360 con 0x7f6a3c0cfc30 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a42359640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3c0d2c20 con 0x7f6a3c0d2120 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c1ea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c1ea060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42820/0 (socket says 172.21.15.156:42820) 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 -- 172.21.15.156:0/939950894 learned_addr learned my addr 172.21.15.156:0/939950894 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a3bfff640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a3c0cfc30 0x7f6a3c0c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a3b7fe640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a3c0d2120 0x7f6a3c1e91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 -- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a3c0d2120 msgr2=0x7f6a3c1e91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a3c0d2120 0x7f6a3c1e91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 -- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a3c0cfc30 msgr2=0x7f6a3c0c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a3c0cfc30 0x7f6a3c0c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.736+0000 7f6a408f6640 1 -- 172.21.15.156:0/939950894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c0ce160 con 0x7f6a3c15e920 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.737+0000 7f6a408f6640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c1ea060 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6a34007e40 tx=0x7f6a3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.737+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34014620 con 0x7f6a3c15e920 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.737+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a34019070 con 0x7f6a3c15e920 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.737+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34014a20 con 0x7f6a3c15e920 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.737+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c0cc060 con 0x7f6a3c15e920 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.737+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c0ca4a0 con 0x7f6a3c15e920 2024-03-29T04:09:27.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.738+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3c0a2480 con 0x7f6a3c15e920 2024-03-29T04:09:27.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.738+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6a34014bc0 con 0x7f6a3c15e920 2024-03-29T04:09:27.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.739+0000 7f6a38ff9640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6a2806c560 0x7f6a2806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:27.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.739+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6a3408be10 con 0x7f6a3c15e920 2024-03-29T04:09:27.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.739+0000 7f6a3bfff640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6a2806c560 0x7f6a2806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:27.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.739+0000 7f6a3bfff640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6a2806c560 0x7f6a2806ea20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c036000 tx=0x7f6a2c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:27.727 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.747+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6a34059de0 con 0x7f6a3c15e920 2024-03-29T04:09:27.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.928+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6a3c04f9d0 con 0x7f6a3c15e920 2024-03-29T04:09:27.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.928+0000 7f6a38ff9640 1 -- 172.21.15.156:0/939950894 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f6a3405e0e0 con 0x7f6a3c15e920 2024-03-29T04:09:27.908 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6a2806c560 msgr2=0x7f6a2806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6a2806c560 0x7f6a2806ea20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c036000 tx=0x7f6a2c03b040 comp rx=0 tx=0).stop 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 msgr2=0x7f6a3c1ea060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c1ea060 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6a34007e40 tx=0x7f6a3400e4b0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 shutdown_connections 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6a2806c560 0x7f6a2806ea20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a3c15e920 0x7f6a3c1ea060 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a3c0d2120 0x7f6a3c1e91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 --2- 172.21.15.156:0/939950894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a3c0cfc30 0x7f6a3c0c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:27.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 >> 172.21.15.156:0/939950894 conn(0x7f6a3c09d7a0 msgr2=0x7f6a3c0b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:27.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 shutdown_connections 2024-03-29T04:09:27.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:27.930+0000 7f6a42359640 1 -- 172.21.15.156:0/939950894 wait complete. 2024-03-29T04:09:27.921 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:28.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 Processor -- start 2024-03-29T04:09:28.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 -- start start 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 0x7f15e415c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f15e415ceb0 0x7f15e40d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e415d960 con 0x7f15e4071020 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f15e413f7f0 con 0x7f15e415ceb0 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.056+0000 7f15eb06a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f15e415d3a0 con 0x7f15e4141ca0 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e8e06640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f15e4071020 0x7f15e4071400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58046/0 (socket says 172.21.15.156:58046) 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e8e06640 1 -- 172.21.15.156:0/1193125162 learned_addr learned my addr 172.21.15.156:0/1193125162 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182370977 0 0) 0x7f15e415d960 con 0x7f15e4071020 2024-03-29T04:09:28.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e3fff640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 0x7f15e415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e9607640 1 --2- 172.21.15.156:0/1193125162 >> v2:172.21.15.203:3300/0 conn(0x7f15e415ceb0 0x7f15e40d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15cc003520 con 0x7f15e4071020 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3166946312 0 0) 0x7f15cc003520 con 0x7f15e4071020 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 msgr2=0x7f15e415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 0x7f15e415c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 >> v2:172.21.15.203:3300/0 conn(0x7f15e415ceb0 msgr2=0x7f15e40d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 --2- 172.21.15.156:0/1193125162 >> v2:172.21.15.203:3300/0 conn(0x7f15e415ceb0 0x7f15e40d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e415d680 con 0x7f15e4071020 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f15dc0019f0 con 0x7f15e4071020 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 >> v1:172.21.15.156:6789/0 conn(0x7f15e4071020 legacy=0x7f15e4071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 0x7f15cc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15cc003f30 0x7f15cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc004d40 0x7f15cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15e413f7f0 con 0x7f15cc004d40 2024-03-29T04:09:28.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15e415d3a0 con 0x7f15cc003f30 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f15cc00b9c0 con 0x7f15e4141ca0 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e9607640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc004d40 0x7f15cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.057+0000 7f15e8e06640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 0x7f15cc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e3fff640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15cc003f30 0x7f15cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e9607640 1 -- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 msgr2=0x7f15cc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e9607640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e4141ca0 0x7f15cc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e9607640 1 -- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15cc003f30 msgr2=0x7f15cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e9607640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15cc003f30 0x7f15cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e9607640 1 -- 172.21.15.156:0/1193125162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc003520 con 0x7f15cc004d40 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.058+0000 7f15e9607640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc004d40 0x7f15cc0051c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009a90 tx=0x7f15cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a575e82029e03029 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4039070 con 0x7f15cc004d40 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d402c6b0 con 0x7f15cc004d40 2024-03-29T04:09:28.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15e37fe640 1 -- 172.21.15.156:0/1193125162 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4009690 con 0x7f15cc004d40 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15eb06a640 1 -- 172.21.15.156:0/1193125162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc004d40 msgr2=0x7f15cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1193125162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc004d40 0x7f15cc0051c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009a90 tx=0x7f15cc005830 comp rx=0 tx=0).stop 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15e9607640 1 -- 172.21.15.156:0/1193125162 reap_dead start 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15eb06a640 1 -- 172.21.15.156:0/1193125162 shutdown_connections 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15eb06a640 1 -- 172.21.15.156:0/1193125162 >> 172.21.15.156:0/1193125162 conn(0x7f15e40b3eb0 msgr2=0x7f15e40b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15eb06a640 1 -- 172.21.15.156:0/1193125162 shutdown_connections 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.059+0000 7f15eb06a640 1 -- 172.21.15.156:0/1193125162 wait complete. 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.060+0000 7f15eb06a640 1 Processor -- start 2024-03-29T04:09:28.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.060+0000 7f15eb06a640 1 -- start start 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15eb06a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413e1e0 0x7f15e41f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15eb06a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 0x7f15e40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15eb06a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e413edc0 0x7f15e40d2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15eb06a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15e415c970 con 0x7f15e413e890 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15eb06a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15e40d1280 con 0x7f15e413e1e0 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15eb06a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f15e415caf0 con 0x7f15e413edc0 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 0x7f15e40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 0x7f15e40d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42846/0 (socket says 172.21.15.156:42846) 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15e3fff640 1 -- 172.21.15.156:0/1924055144 learned_addr learned my addr 172.21.15.156:0/1924055144 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15e8e06640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413e1e0 0x7f15e41f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.061+0000 7f15e9607640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e413edc0 0x7f15e40d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e3fff640 1 -- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e413edc0 msgr2=0x7f15e40d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e3fff640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e413edc0 0x7f15e40d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e3fff640 1 -- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413e1e0 msgr2=0x7f15e41f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e3fff640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413e1e0 0x7f15e41f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.044 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e3fff640 1 -- 172.21.15.156:0/1924055144 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e4145400 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e3fff640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 0x7f15e40d1df0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f15d8011960 tx=0x7f15d8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d800f9d0 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d800fb70 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8014440 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e41456e0 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.062+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e41f5b00 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.063+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e4070e10 con 0x7f15e413e890 2024-03-29T04:09:28.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.063+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f15d8012040 con 0x7f15e413e890 2024-03-29T04:09:28.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.064+0000 7f15e17fa640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f15d006c530 0x7f15d006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.064+0000 7f15e8e06640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f15d006c530 0x7f15d006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.064+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f15d808baf0 con 0x7f15e413e890 2024-03-29T04:09:28.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.065+0000 7f15e8e06640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f15d006c530 0x7f15d006e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f15dc004bc0 tx=0x7f15dc006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.068+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f15d8019030 con 0x7f15e413e890 2024-03-29T04:09:28.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.249+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15e40d2ce0 con 0x7f15e413e890 2024-03-29T04:09:28.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.249+0000 7f15e17fa640 1 -- 172.21.15.156:0/1924055144 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f15d805a1a0 con 0x7f15e413e890 2024-03-29T04:09:28.229 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f15d006c530 msgr2=0x7f15d006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f15d006c530 0x7f15d006e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f15dc004bc0 tx=0x7f15dc006040 comp rx=0 tx=0).stop 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 msgr2=0x7f15e40d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 0x7f15e40d1df0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f15d8011960 tx=0x7f15d8011d60 comp rx=0 tx=0).stop 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 shutdown_connections 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f15d006c530 0x7f15d006e9f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f15e413edc0 0x7f15e40d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e413e890 0x7f15e40d1df0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 --2- 172.21.15.156:0/1924055144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413e1e0 0x7f15e41f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 >> 172.21.15.156:0/1924055144 conn(0x7f15e40b3eb0 msgr2=0x7f15e40b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:28.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 shutdown_connections 2024-03-29T04:09:28.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.251+0000 7f15eb06a640 1 -- 172.21.15.156:0/1924055144 wait complete. 2024-03-29T04:09:28.242 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:28.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.366+0000 7f39dc5d4640 1 Processor -- start 2024-03-29T04:09:28.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.366+0000 7f39dc5d4640 1 -- start start 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.367+0000 7f39dc5d4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f39d41415d0 0x7f39d415eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.367+0000 7f39dc5d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d40b3dd0 0x7f39d40b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.367+0000 7f39dc5d4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39d4162130 con 0x7f39d4144650 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.367+0000 7f39dc5d4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f39d40b4720 con 0x7f39d41415d0 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.367+0000 7f39dc5d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f39d41422c0 con 0x7f39d40b3dd0 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39d9b6f640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f39d4144650 0x7f39d4141bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58060/0 (socket says 172.21.15.156:58060) 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39d9b6f640 1 -- 172.21.15.156:0/827732216 learned_addr learned my addr 172.21.15.156:0/827732216 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39dab71640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d40b3dd0 0x7f39d40b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39da370640 1 --2- 172.21.15.156:0/827732216 >> v2:172.21.15.203:3300/0 conn(0x7f39d41415d0 0x7f39d415eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39d936e640 1 -- 172.21.15.156:0/827732216 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2782352663 0 0) 0x7f39d4162130 con 0x7f39d4144650 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39d936e640 1 -- 172.21.15.156:0/827732216 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39c0003520 con 0x7f39d4144650 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39d936e640 1 -- 172.21.15.156:0/827732216 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364910947 0 0) 0x7f39c0003520 con 0x7f39d4144650 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39da370640 1 -- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d40b3dd0 msgr2=0x7f39d40b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39da370640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d40b3dd0 0x7f39d40b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39da370640 1 -- 172.21.15.156:0/827732216 >> v1:172.21.15.156:6789/0 conn(0x7f39d4144650 legacy=0x7f39d4141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39da370640 1 -- 172.21.15.156:0/827732216 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d415f0f0 con 0x7f39d41415d0 2024-03-29T04:09:28.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.368+0000 7f39dab71640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d40b3dd0 0x7f39d40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.369+0000 7f39da370640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d41415d0 0x7f39d415eaa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f39d00097a0 tx=0x7f39d002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=228c9131eaeef38b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.369+0000 7f39d936e640 1 -- 172.21.15.156:0/827732216 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0037070 con 0x7f39d41415d0 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.369+0000 7f39d936e640 1 -- 172.21.15.156:0/827732216 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d0030510 con 0x7f39d41415d0 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.369+0000 7f39d936e640 1 -- 172.21.15.156:0/827732216 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4001690 con 0x7f39d41415d0 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 -- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d41415d0 msgr2=0x7f39d415eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d41415d0 0x7f39d415eaa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f39d00097a0 tx=0x7f39d002b790 comp rx=0 tx=0).stop 2024-03-29T04:09:28.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 -- 172.21.15.156:0/827732216 shutdown_connections 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d40b3dd0 0x7f39d40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/827732216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d41415d0 0x7f39d415eaa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 -- 172.21.15.156:0/827732216 >> 172.21.15.156:0/827732216 conn(0x7f39d40b7660 msgr2=0x7f39d40b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 -- 172.21.15.156:0/827732216 shutdown_connections 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.370+0000 7f39dc5d4640 1 -- 172.21.15.156:0/827732216 wait complete. 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.371+0000 7f39dc5d4640 1 Processor -- start 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.371+0000 7f39dc5d4640 1 -- start start 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 0x7f39d41456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d41415d0 0x7f39d4142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d4144650 0x7f39d4143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f39d4144ae0 con 0x7f39d40b3dd0 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f39d415f6b0 con 0x7f39d4144650 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f39d415f990 con 0x7f39d41415d0 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 0x7f39d41456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 0x7f39d41456d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42852/0 (socket says 172.21.15.156:42852) 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 -- 172.21.15.156:0/2685810981 learned_addr learned my addr 172.21.15.156:0/2685810981 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 -- 172.21.15.156:0/2685810981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d41415d0 msgr2=0x7f39d4142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d41415d0 0x7f39d4142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 -- 172.21.15.156:0/2685810981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d4144650 msgr2=0x7f39d4143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d4144650 0x7f39d4143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 -- 172.21.15.156:0/2685810981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d4146390 con 0x7f39d40b3dd0 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39da370640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 0x7f39d41456d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f39d002b7d0 tx=0x7f39d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0041020 con 0x7f39d40b3dd0 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d003c870 con 0x7f39d40b3dd0 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0037040 con 0x7f39d40b3dd0 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d4146670 con 0x7f39d40b3dd0 2024-03-29T04:09:28.354 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.372+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d4146c30 con 0x7f39d40b3dd0 2024-03-29T04:09:28.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.373+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d404f9d0 con 0x7f39d40b3dd0 2024-03-29T04:09:28.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.374+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f39d0044020 con 0x7f39d40b3dd0 2024-03-29T04:09:28.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.375+0000 7f39caffd640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f39b806c530 0x7f39b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.375+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f39d00acf80 con 0x7f39d40b3dd0 2024-03-29T04:09:28.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.375+0000 7f39d9b6f640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f39b806c530 0x7f39b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.375+0000 7f39d9b6f640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f39b806c530 0x7f39b806e9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f39c4001eb0 tx=0x7f39c4001a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.383+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f39d004b030 con 0x7f39d40b3dd0 2024-03-29T04:09:28.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.559+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f39d40d2f60 con 0x7f39d40b3dd0 2024-03-29T04:09:28.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.559+0000 7f39caffd640 1 -- 172.21.15.156:0/2685810981 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f39d007b630 con 0x7f39d40b3dd0 2024-03-29T04:09:28.539 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f39b806c530 msgr2=0x7f39b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f39b806c530 0x7f39b806e9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f39c4001eb0 tx=0x7f39c4001a20 comp rx=0 tx=0).stop 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 msgr2=0x7f39d41456d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 0x7f39d41456d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f39d002b7d0 tx=0x7f39d00046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 shutdown_connections 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f39b806c530 0x7f39b806e9f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39d4144650 0x7f39d4143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39d41415d0 0x7f39d4142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 --2- 172.21.15.156:0/2685810981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d40b3dd0 0x7f39d41456d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 >> 172.21.15.156:0/2685810981 conn(0x7f39d40b7660 msgr2=0x7f39d4140c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:28.541 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 shutdown_connections 2024-03-29T04:09:28.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.561+0000 7f39dc5d4640 1 -- 172.21.15.156:0/2685810981 wait complete. 2024-03-29T04:09:28.552 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:28.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.675+0000 7fb0ba3ba640 1 Processor -- start 2024-03-29T04:09:28.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.676+0000 7fb0ba3ba640 1 -- start start 2024-03-29T04:09:28.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.676+0000 7fb0ba3ba640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 0x7fb0b406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.676+0000 7fb0ba3ba640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0b40d2df0 con 0x7fb0b415e920 2024-03-29T04:09:28.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.676+0000 7fb0ba3ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0b40cf190 con 0x7fb0b406cb10 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.676+0000 7fb0ba3ba640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b40cf7a0 con 0x7fb0b40d2120 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb0b415e920 0x7fb0b40d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58074/0 (socket says 172.21.15.156:58074) 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b3fff640 1 -- 172.21.15.156:0/339809714 learned_addr learned my addr 172.21.15.156:0/339809714 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010119224 0 0) 0x7fb0b40d2df0 con 0x7fb0b415e920 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 0x7fb0b406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb09c003520 con 0x7fb0b415e920 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2450392883 0 0) 0x7fb09c003520 con 0x7fb0b415e920 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 msgr2=0x7fb0b406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 0x7fb0b406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 >> v1:172.21.15.203:6789/0 conn(0x7fb0b406cb10 legacy=0x7fb0b40cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b40d3080 con 0x7fb0b415e920 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fb0a4001670 con 0x7fb0b415e920 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 >> v1:172.21.15.156:6789/0 conn(0x7fb0b415e920 legacy=0x7fb0b40d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 0x7fb09c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb09c003f20 0x7fb09c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb09c004d40 0x7fb09c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b40cf7a0 con 0x7fb09c003f20 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c00ba20 con 0x7fb09c004d40 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb09c00bc20 con 0x7fb0b40d2120 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb09c003f20 0x7fb09c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b3fff640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 0x7fb09c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.677+0000 7fb0b8957640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb09c004d40 0x7fb09c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.678+0000 7fb0b37fe640 1 -- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 msgr2=0x7fb09c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.678+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0b40d2120 0x7fb09c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.678+0000 7fb0b37fe640 1 -- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb09c004d40 msgr2=0x7fb09c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.678+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb09c004d40 0x7fb09c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.678+0000 7fb0b37fe640 1 -- 172.21.15.156:0/339809714 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c003520 con 0x7fb09c003f20 2024-03-29T04:09:28.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.678+0000 7fb0b8957640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb09c004d40 0x7fb09c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb09c003f20 0x7fb09c004800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009aa0 tx=0x7fb09c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77cabb1b5e5aeb92 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8039070 con 0x7fb09c003f20 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a802c600 con 0x7fb09c003f20 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0b2ffd640 1 -- 172.21.15.156:0/339809714 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8009670 con 0x7fb09c003f20 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/339809714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb09c003f20 msgr2=0x7fb09c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/339809714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb09c003f20 0x7fb09c004800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009aa0 tx=0x7fb09c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0b8957640 1 -- 172.21.15.156:0/339809714 reap_dead start 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/339809714 shutdown_connections 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.679+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/339809714 >> 172.21.15.156:0/339809714 conn(0x7fb0b409d7a0 msgr2=0x7fb0b40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.680+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/339809714 shutdown_connections 2024-03-29T04:09:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.680+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/339809714 wait complete. 2024-03-29T04:09:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.681+0000 7fb0ba3ba640 1 Processor -- start 2024-03-29T04:09:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.681+0000 7fb0ba3ba640 1 -- start start 2024-03-29T04:09:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.681+0000 7fb0ba3ba640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb09c003f20 0x7fb0b4159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 0x7fb0b4161220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b409e210 0x7fb0b409e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b40d34e0 con 0x7fb0b4160da0 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b40d3360 con 0x7fb0b409e210 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b40d0340 con 0x7fb09c003f20 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 0x7fb0b4161220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 0x7fb0b4161220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42878/0 (socket says 172.21.15.156:42878) 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 -- 172.21.15.156:0/3551820887 learned_addr learned my addr 172.21.15.156:0/3551820887 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b8957640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b409e210 0x7fb0b409e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b3fff640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb09c003f20 0x7fb0b4159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 -- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb09c003f20 msgr2=0x7fb0b4159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb09c003f20 0x7fb0b4159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 -- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b409e210 msgr2=0x7fb0b409e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b409e210 0x7fb0b409e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 -- 172.21.15.156:0/3551820887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b41f5b10 con 0x7fb0b4160da0 2024-03-29T04:09:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b37fe640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 0x7fb0b4161220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009b00 tx=0x7fb0a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8044020 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a803f670 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b409eb90 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8039040 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.682+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b409f140 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.683+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b40a2480 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.684+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb0a8047020 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.684+0000 7fb0b0ff9640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0a006c530 0x7fb0a006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.684+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb0a80ad390 con 0x7fb0b4160da0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.684+0000 7fb0b3fff640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0a006c530 0x7fb0a006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:28.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.685+0000 7fb0b3fff640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0a006c530 0x7fb0a006e9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4001550 tx=0x7fb0a4000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:28.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.689+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb0a804e070 con 0x7fb0b4160da0 2024-03-29T04:09:28.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.881+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb0b40d2dc0 con 0x7fb0b4160da0 2024-03-29T04:09:28.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.882+0000 7fb0b0ff9640 1 -- 172.21.15.156:0/3551820887 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fb0a807b930 con 0x7fb0b4160da0 2024-03-29T04:09:28.861 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:28.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0a006c530 msgr2=0x7fb0a006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0a006c530 0x7fb0a006e9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4001550 tx=0x7fb0a4000ed0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 msgr2=0x7fb0b4161220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 0x7fb0b4161220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009b00 tx=0x7fb0a8004730 comp rx=0 tx=0).stop 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 shutdown_connections 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0a006c530 0x7fb0a006e9f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b409e210 0x7fb0b409e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b4160da0 0x7fb0b4161220 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 --2- 172.21.15.156:0/3551820887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb09c003f20 0x7fb0b4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 >> 172.21.15.156:0/3551820887 conn(0x7fb0b409d7a0 msgr2=0x7fb0b415f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 shutdown_connections 2024-03-29T04:09:28.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:28.884+0000 7fb0ba3ba640 1 -- 172.21.15.156:0/3551820887 wait complete. 2024-03-29T04:09:28.874 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-03-29T04:09:28.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:28.874 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-29T04:09:28.903 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:28.903 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:09:28.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:28.903 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T04:09:28.967 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:09:28.967 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:28.986 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:09:28.986 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:29.044 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:29.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.166+0000 7fd532924640 1 Processor -- start 2024-03-29T04:09:29.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.166+0000 7fd532924640 1 -- start start 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd532924640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd52c0d42c0 0x7fd52c0d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd532924640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c06cbb0 0x7fd52c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd532924640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd52c0d4c80 con 0x7fd52c15e730 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd532924640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd52c06c720 con 0x7fd52c0d42c0 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd532924640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd52c0cfa20 con 0x7fd52c06cbb0 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd52c0d42c0 0x7fd52c0d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd52c0d42c0 0x7fd52c0d4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42884/0 (socket says 172.21.15.156:42884) 2024-03-29T04:09:29.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 -- 172.21.15.156:0/681263405 learned_addr learned my addr 172.21.15.156:0/681263405 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:29.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd530ec1640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c06cbb0 0x7fd52c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 -- 172.21.15.156:0/681263405 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c06cbb0 msgr2=0x7fd52c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:29.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c06cbb0 0x7fd52c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 -- 172.21.15.156:0/681263405 >> v1:172.21.15.203:6789/0 conn(0x7fd52c15e730 legacy=0x7fd52c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd52b7fe640 1 -- 172.21.15.156:0/681263405 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c0cfd00 con 0x7fd52c0d42c0 2024-03-29T04:09:29.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.167+0000 7fd530ec1640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c06cbb0 0x7fd52c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:29.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.168+0000 7fd52b7fe640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0d4740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd520009850 tx=0x7fd52002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b99973e92e1a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.168+0000 7fd52affd640 1 -- 172.21.15.156:0/681263405 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520004230 con 0x7fd52c0d42c0 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.168+0000 7fd52affd640 1 -- 172.21.15.156:0/681263405 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5200043d0 con 0x7fd52c0d42c0 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.168+0000 7fd52affd640 1 -- 172.21.15.156:0/681263405 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520003810 con 0x7fd52c0d42c0 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 -- 172.21.15.156:0/681263405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 msgr2=0x7fd52c0d4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0d4740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd520009850 tx=0x7fd52002b960 comp rx=0 tx=0).stop 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 -- 172.21.15.156:0/681263405 shutdown_connections 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c06cbb0 0x7fd52c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 --2- 172.21.15.156:0/681263405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0d4740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 -- 172.21.15.156:0/681263405 >> 172.21.15.156:0/681263405 conn(0x7fd52c08bfd0 msgr2=0x7fd52c09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 -- 172.21.15.156:0/681263405 shutdown_connections 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.169+0000 7fd532924640 1 -- 172.21.15.156:0/681263405 wait complete. 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.170+0000 7fd532924640 1 Processor -- start 2024-03-29T04:09:29.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.170+0000 7fd532924640 1 -- start start 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd532924640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd52c06cbb0 0x7fd52c153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd532924640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd532924640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c15e730 0x7fd52c0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd532924640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c0d2220 con 0x7fd52c0d42c0 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd532924640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c0d02c0 con 0x7fd52c06cbb0 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd532924640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd52c15eb10 con 0x7fd52c15e730 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42886/0 (socket says 172.21.15.156:42886) 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 -- 172.21.15.156:0/741897956 learned_addr learned my addr 172.21.15.156:0/741897956 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:29.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52bfff640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd52c06cbb0 0x7fd52c153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd530ec1640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c15e730 0x7fd52c0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 -- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c15e730 msgr2=0x7fd52c0c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c15e730 0x7fd52c0c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 -- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd52c06cbb0 msgr2=0x7fd52c153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd52c06cbb0 0x7fd52c153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 -- 172.21.15.156:0/741897956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c0cc8e0 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.171+0000 7fd52b7fe640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0c8630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd52002b9a0 tx=0x7fd5200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.172+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520048070 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.172+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd520003bd0 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.172+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520006040 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.172+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c0ccb80 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.172+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1579a0 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.173+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd52c068d20 con 0x7fd52c0d42c0 2024-03-29T04:09:29.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.173+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd520004330 con 0x7fd52c0d42c0 2024-03-29T04:09:29.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.174+0000 7fd528ff9640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd51806c530 0x7fd51806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.174+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd5200ad160 con 0x7fd52c0d42c0 2024-03-29T04:09:29.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.174+0000 7fd52bfff640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd51806c530 0x7fd51806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.174+0000 7fd52bfff640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd51806c530 0x7fd51806e9f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd51c007fd0 tx=0x7fd51c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.178+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd52c068d20 con 0x7fd52c0d42c0 2024-03-29T04:09:29.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.363+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd52c068610 con 0x7fd52c0d42c0 2024-03-29T04:09:29.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.365+0000 7fd528ff9640 1 -- 172.21.15.156:0/741897956 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fd520034020 con 0x7fd52c0d42c0 2024-03-29T04:09:29.344 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:29.344 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:29.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.366+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd51806c530 msgr2=0x7fd51806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.366+0000 7fd532924640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd51806c530 0x7fd51806e9f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd51c007fd0 tx=0x7fd51c009040 comp rx=0 tx=0).stop 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.366+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 msgr2=0x7fd52c0c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.366+0000 7fd532924640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0c8630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd52002b9a0 tx=0x7fd5200046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 shutdown_connections 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd51806c530 0x7fd51806e9f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd52c15e730 0x7fd52c0c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c0d42c0 0x7fd52c0c8630 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 --2- 172.21.15.156:0/741897956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd52c06cbb0 0x7fd52c153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 >> 172.21.15.156:0/741897956 conn(0x7fd52c08bfd0 msgr2=0x7fd52c0b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 shutdown_connections 2024-03-29T04:09:29.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.367+0000 7fd532924640 1 -- 172.21.15.156:0/741897956 wait complete. 2024-03-29T04:09:29.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:29.358 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:29.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 Processor -- start 2024-03-29T04:09:29.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 -- start start 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d840d0350 0x7f7d8415c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7d840d0730 0x7f7d8406d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7d8406dba0 0x7f7d8415eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8415c8d0 con 0x7f7d8406dba0 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8415bf00 con 0x7f7d840d0730 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d8afba640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8415b8d0 con 0x7f7d840d0350 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.484+0000 7f7d89557640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7d8406dba0 0x7f7d8415eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7d8406dba0 0x7f7d8415eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42894/0 (socket says 172.21.15.156:42894) 2024-03-29T04:09:29.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 -- 172.21.15.156:0/2752898950 learned_addr learned my addr 172.21.15.156:0/2752898950 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d88d56640 1 --2- 172.21.15.156:0/2752898950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d840d0350 0x7f7d8415c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d83fff640 1 --2- 172.21.15.156:0/2752898950 >> v2:172.21.15.203:3300/0 conn(0x7f7d840d0730 0x7f7d8406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 -- 172.21.15.156:0/2752898950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d840d0350 msgr2=0x7f7d8415c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 --2- 172.21.15.156:0/2752898950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d840d0350 0x7f7d8415c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 -- 172.21.15.156:0/2752898950 >> v2:172.21.15.203:3300/0 conn(0x7f7d840d0730 msgr2=0x7f7d8406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 --2- 172.21.15.156:0/2752898950 >> v2:172.21.15.203:3300/0 conn(0x7f7d840d0730 0x7f7d8406d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.485+0000 7f7d89557640 1 -- 172.21.15.156:0/2752898950 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8406e120 con 0x7f7d8406dba0 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d89557640 1 --2- 172.21.15.156:0/2752898950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d8406dba0 0x7f7d8415eec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406c650 tx=0x7f7d7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb3b89ae16b7272 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d837fe640 1 -- 172.21.15.156:0/2752898950 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c02ca70 con 0x7f7d8406dba0 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d837fe640 1 -- 172.21.15.156:0/2752898950 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7c02cc10 con 0x7f7d8406dba0 2024-03-29T04:09:29.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d837fe640 1 -- 172.21.15.156:0/2752898950 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c0343f0 con 0x7f7d8406dba0 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 -- 172.21.15.156:0/2752898950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d8406dba0 msgr2=0x7f7d8415eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 --2- 172.21.15.156:0/2752898950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d8406dba0 0x7f7d8415eec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406c650 tx=0x7f7d7c02bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 -- 172.21.15.156:0/2752898950 shutdown_connections 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 --2- 172.21.15.156:0/2752898950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d8406dba0 0x7f7d8415eec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 --2- 172.21.15.156:0/2752898950 >> v2:172.21.15.203:3300/0 conn(0x7f7d840d0730 0x7f7d8406d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 --2- 172.21.15.156:0/2752898950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d840d0350 0x7f7d8415c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 -- 172.21.15.156:0/2752898950 >> 172.21.15.156:0/2752898950 conn(0x7f7d840b1f30 msgr2=0x7f7d840b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 -- 172.21.15.156:0/2752898950 shutdown_connections 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.486+0000 7f7d8afba640 1 -- 172.21.15.156:0/2752898950 wait complete. 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.487+0000 7f7d8afba640 1 Processor -- start 2024-03-29T04:09:29.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.487+0000 7f7d8afba640 1 -- start start 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d8afba640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d8406dba0 0x7f7d840b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d8afba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d840d0350 0x7f7d84152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d8afba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 0x7f7d8414fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d8afba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84073700 con 0x7f7d840d0730 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d8afba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8415bb90 con 0x7f7d840d0350 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d8afba640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7d840d0bb0 con 0x7f7d8406dba0 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d89557640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 0x7f7d8414fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d89557640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 0x7f7d8414fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42908/0 (socket says 172.21.15.156:42908) 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.488+0000 7f7d89557640 1 -- 172.21.15.156:0/3856625969 learned_addr learned my addr 172.21.15.156:0/3856625969 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d88d56640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d8406dba0 0x7f7d840b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d83fff640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d840d0350 0x7f7d84152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d89557640 1 -- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d8406dba0 msgr2=0x7f7d840b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d89557640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d8406dba0 0x7f7d840b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d89557640 1 -- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d840d0350 msgr2=0x7f7d84152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d89557640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d840d0350 0x7f7d84152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d89557640 1 -- 172.21.15.156:0/3856625969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d841529d0 con 0x7f7d840d0730 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d89557640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 0x7f7d8414fa50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036040 tx=0x7f7d7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c043020 con 0x7f7d840d0730 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7c03e450 con 0x7f7d840d0730 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d840b0bb0 con 0x7f7d840d0730 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c009510 con 0x7f7d840d0730 2024-03-29T04:09:29.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.489+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d840b1170 con 0x7f7d840d0730 2024-03-29T04:09:29.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.490+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d8409e3b0 con 0x7f7d840d0730 2024-03-29T04:09:29.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.490+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7d7c046020 con 0x7f7d840d0730 2024-03-29T04:09:29.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.491+0000 7f7d817fa640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7d6806c530 0x7f7d6806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.491+0000 7f7d88d56640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7d6806c530 0x7f7d6806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.491+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7d7c0add50 con 0x7f7d840d0730 2024-03-29T04:09:29.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.491+0000 7f7d88d56640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7d6806c530 0x7f7d6806e9f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d84155b10 tx=0x7f7d78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.495+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7d7c0b1030 con 0x7f7d840d0730 2024-03-29T04:09:29.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.671+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d8404f9d0 con 0x7f7d840d0730 2024-03-29T04:09:29.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.671+0000 7f7d817fa640 1 -- 172.21.15.156:0/3856625969 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f7d7c04d030 con 0x7f7d840d0730 2024-03-29T04:09:29.651 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7d6806c530 msgr2=0x7f7d6806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7d6806c530 0x7f7d6806e9f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d84155b10 tx=0x7f7d78008040 comp rx=0 tx=0).stop 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 msgr2=0x7f7d8414fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 0x7f7d8414fa50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036040 tx=0x7f7d7c004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 shutdown_connections 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7d6806c530 0x7f7d6806e9f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d840d0730 0x7f7d8414fa50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d840d0350 0x7f7d84152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 --2- 172.21.15.156:0/3856625969 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d8406dba0 0x7f7d840b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 >> 172.21.15.156:0/3856625969 conn(0x7f7d840b1f30 msgr2=0x7f7d840af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:29.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 shutdown_connections 2024-03-29T04:09:29.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.673+0000 7f7d8afba640 1 -- 172.21.15.156:0/3856625969 wait complete. 2024-03-29T04:09:29.664 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.789+0000 7fdd8faa7640 1 Processor -- start 2024-03-29T04:09:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.790+0000 7fdd8faa7640 1 -- start start 2024-03-29T04:09:29.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.790+0000 7fdd8faa7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdd8815e920 0x7fdd880d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.790+0000 7fdd8faa7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd880cfc30 0x7fdd880d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.790+0000 7fdd8faa7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd880cf190 con 0x7fdd880d2120 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.790+0000 7fdd8faa7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdd880cf7a0 con 0x7fdd880cfc30 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.790+0000 7fdd8faa7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdd880d3080 con 0x7fdd8815e920 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd880cfc30 0x7fdd880d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd880cfc30 0x7fdd880d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42912/0 (socket says 172.21.15.156:42912) 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 -- 172.21.15.156:0/778957414 learned_addr learned my addr 172.21.15.156:0/778957414 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8d843640 1 --2- 172.21.15.156:0/778957414 >> v2:172.21.15.203:3300/0 conn(0x7fdd8815e920 0x7fdd880d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 -- 172.21.15.156:0/778957414 >> v2:172.21.15.203:3300/0 conn(0x7fdd8815e920 msgr2=0x7fdd880d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 --2- 172.21.15.156:0/778957414 >> v2:172.21.15.203:3300/0 conn(0x7fdd8815e920 0x7fdd880d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 -- 172.21.15.156:0/778957414 >> v1:172.21.15.203:6789/0 conn(0x7fdd880d2120 legacy=0x7fdd8806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.791+0000 7fdd8e044640 1 -- 172.21.15.156:0/778957414 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd880cf430 con 0x7fdd880cfc30 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8e044640 1 --2- 172.21.15.156:0/778957414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880cfc30 0x7fdd880d00c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdd880b7040 tx=0x7fdd8002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0d38a8c9d14bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8c841640 1 -- 172.21.15.156:0/778957414 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8002cb20 con 0x7fdd880cfc30 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8c841640 1 -- 172.21.15.156:0/778957414 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8002ccc0 con 0x7fdd880cfc30 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8c841640 1 -- 172.21.15.156:0/778957414 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80034610 con 0x7fdd880cfc30 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 -- 172.21.15.156:0/778957414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880cfc30 msgr2=0x7fdd880d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/778957414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880cfc30 0x7fdd880d00c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdd880b7040 tx=0x7fdd8002bf80 comp rx=0 tx=0).stop 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 -- 172.21.15.156:0/778957414 shutdown_connections 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/778957414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880cfc30 0x7fdd880d00c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/778957414 >> v2:172.21.15.203:3300/0 conn(0x7fdd8815e920 0x7fdd880d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 -- 172.21.15.156:0/778957414 >> 172.21.15.156:0/778957414 conn(0x7fdd8809d7a0 msgr2=0x7fdd880d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 -- 172.21.15.156:0/778957414 shutdown_connections 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.792+0000 7fdd8faa7640 1 -- 172.21.15.156:0/778957414 wait complete. 2024-03-29T04:09:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.794+0000 7fdd8faa7640 1 Processor -- start 2024-03-29T04:09:29.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.794+0000 7fdd8faa7640 1 -- start start 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8faa7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd880cfc30 0x7fdd8814da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8faa7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 0x7fdd8815ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8faa7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8815e920 0x7fdd8815f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8faa7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880d34e0 con 0x7fdd880d2120 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8faa7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880d3360 con 0x7fdd880cfc30 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8faa7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdd880d2c20 con 0x7fdd8815e920 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 0x7fdd8815ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 0x7fdd8815ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42916/0 (socket says 172.21.15.156:42916) 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 -- 172.21.15.156:0/55994706 learned_addr learned my addr 172.21.15.156:0/55994706 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d843640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd880cfc30 0x7fdd8814da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8e044640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8815e920 0x7fdd8815f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 -- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8815e920 msgr2=0x7fdd8815f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8815e920 0x7fdd8815f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 -- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd880cfc30 msgr2=0x7fdd8814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd880cfc30 0x7fdd8814da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 -- 172.21.15.156:0/55994706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd880c9b00 con 0x7fdd880d2120 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.795+0000 7fdd8d042640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 0x7fdd8815ed00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c430 tx=0x7fdd7800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.796+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd780052d0 con 0x7fdd880d2120 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.796+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd78005490 con 0x7fdd880d2120 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.796+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd880c9de0 con 0x7fdd880d2120 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.796+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78007260 con 0x7fdd880d2120 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.796+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd880ca3a0 con 0x7fdd880d2120 2024-03-29T04:09:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.797+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd880cbee0 con 0x7fdd880d2120 2024-03-29T04:09:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.797+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdd78007400 con 0x7fdd880d2120 2024-03-29T04:09:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.798+0000 7fdd7e7fc640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd6c06c530 0x7fdd6c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.798+0000 7fdd8d843640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd6c06c530 0x7fdd6c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.798+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdd7808ac10 con 0x7fdd880d2120 2024-03-29T04:09:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.798+0000 7fdd8d843640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd6c06c530 0x7fdd6c06e9f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd84006fd0 tx=0x7fdd84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:29.781 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:29.802+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdd880cbee0 con 0x7fdd880d2120 2024-03-29T04:09:30.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.144+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdd8804f9d0 con 0x7fdd880d2120 2024-03-29T04:09:30.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.144+0000 7fdd7e7fc640 1 -- 172.21.15.156:0/55994706 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fdd78059230 con 0x7fdd880d2120 2024-03-29T04:09:30.124 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:30.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd6c06c530 msgr2=0x7fdd6c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd6c06c530 0x7fdd6c06e9f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd84006fd0 tx=0x7fdd84008040 comp rx=0 tx=0).stop 2024-03-29T04:09:30.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 msgr2=0x7fdd8815ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 0x7fdd8815ed00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c430 tx=0x7fdd7800c8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 shutdown_connections 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd6c06c530 0x7fdd6c06e9f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8815e920 0x7fdd8815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd880d2120 0x7fdd8815ed00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.146+0000 7fdd8faa7640 1 --2- 172.21.15.156:0/55994706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd880cfc30 0x7fdd8814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.147+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 >> 172.21.15.156:0/55994706 conn(0x7fdd8809d7a0 msgr2=0x7fdd880b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.147+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 shutdown_connections 2024-03-29T04:09:30.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.147+0000 7fdd8faa7640 1 -- 172.21.15.156:0/55994706 wait complete. 2024-03-29T04:09:30.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:30.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.265+0000 7f57ca04a640 1 Processor -- start 2024-03-29T04:09:30.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.265+0000 7f57ca04a640 1 -- start start 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57ca04a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c40d2120 0x7f57c406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57ca04a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f57c406cb10 0x7f57c40cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57ca04a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c40d2df0 con 0x7f57c415e920 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57ca04a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f57c40cf7a0 con 0x7f57c406cb10 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57ca04a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f57c40cf190 con 0x7f57c40d2120 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f57c406cb10 0x7f57c40cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c37fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f57c415e920 0x7f57c40d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58080/0 (socket says 172.21.15.156:58080) 2024-03-29T04:09:30.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c37fe640 1 -- 172.21.15.156:0/3550069758 learned_addr learned my addr 172.21.15.156:0/3550069758 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:30.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c2ffd640 1 --2- 172.21.15.156:0/3550069758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c40d2120 0x7f57c406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c3fff640 1 -- 172.21.15.156:0/3550069758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c40d2120 msgr2=0x7f57c406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:30.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c3fff640 1 --2- 172.21.15.156:0/3550069758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c40d2120 0x7f57c406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c3fff640 1 -- 172.21.15.156:0/3550069758 >> v1:172.21.15.156:6789/0 conn(0x7f57c415e920 legacy=0x7f57c40d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.266+0000 7f57c3fff640 1 -- 172.21.15.156:0/3550069758 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c40d3080 con 0x7f57c406cb10 2024-03-29T04:09:30.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.267+0000 7f57c3fff640 1 --2- 172.21.15.156:0/3550069758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c406cb10 0x7f57c40cfc30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009d90 tx=0x7f57bc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4578ec7bf5f91db4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.267+0000 7f57c27fc640 1 -- 172.21.15.156:0/3550069758 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc039070 con 0x7f57c406cb10 2024-03-29T04:09:30.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.267+0000 7f57c27fc640 1 -- 172.21.15.156:0/3550069758 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc02cc50 con 0x7f57c406cb10 2024-03-29T04:09:30.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.267+0000 7f57c27fc640 1 -- 172.21.15.156:0/3550069758 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc034440 con 0x7f57c406cb10 2024-03-29T04:09:30.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 -- 172.21.15.156:0/3550069758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c406cb10 msgr2=0x7f57c40cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3550069758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c406cb10 0x7f57c40cfc30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009d90 tx=0x7f57bc02bea0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 -- 172.21.15.156:0/3550069758 shutdown_connections 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3550069758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c406cb10 0x7f57c40cfc30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3550069758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c40d2120 0x7f57c406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 -- 172.21.15.156:0/3550069758 >> 172.21.15.156:0/3550069758 conn(0x7f57c409d7a0 msgr2=0x7f57c40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 -- 172.21.15.156:0/3550069758 shutdown_connections 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.268+0000 7f57ca04a640 1 -- 172.21.15.156:0/3550069758 wait complete. 2024-03-29T04:09:30.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.269+0000 7f57ca04a640 1 Processor -- start 2024-03-29T04:09:30.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.269+0000 7f57ca04a640 1 -- start start 2024-03-29T04:09:30.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57ca04a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c406cb10 0x7f57c40b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57ca04a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57c40d2120 0x7f57c40b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57ca04a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 0x7f57c40b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57ca04a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f57c40d34e0 con 0x7f57c415e920 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57ca04a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f57c40d3360 con 0x7f57c40d2120 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57ca04a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f57c40d0170 con 0x7f57c406cb10 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 0x7f57c40b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 0x7f57c40b69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42944/0 (socket says 172.21.15.156:42944) 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 -- 172.21.15.156:0/3811357641 learned_addr learned my addr 172.21.15.156:0/3811357641 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c37fe640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c406cb10 0x7f57c40b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c2ffd640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57c40d2120 0x7f57c40b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 -- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c406cb10 msgr2=0x7f57c40b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c406cb10 0x7f57c40b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 -- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57c40d2120 msgr2=0x7f57c40b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57c40d2120 0x7f57c40b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 -- 172.21.15.156:0/3811357641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c40b7b20 con 0x7f57c415e920 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c2ffd640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57c40d2120 0x7f57c40b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.270+0000 7f57c3fff640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 0x7f57c40b69d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009d90 tx=0x7f57bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.271+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc039050 con 0x7f57c415e920 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.271+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc045070 con 0x7f57c415e920 2024-03-29T04:09:30.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.271+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c40b7e00 con 0x7f57c415e920 2024-03-29T04:09:30.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.271+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc03e610 con 0x7f57c415e920 2024-03-29T04:09:30.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.271+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c40b83c0 con 0x7f57c415e920 2024-03-29T04:09:30.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.272+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f57bc03e7b0 con 0x7f57c415e920 2024-03-29T04:09:30.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.272+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c40a2480 con 0x7f57c415e920 2024-03-29T04:09:30.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.272+0000 7f57abfff640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f57b006c560 0x7f57b006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.272+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f57bc0ae140 con 0x7f57c415e920 2024-03-29T04:09:30.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.272+0000 7f57c37fe640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f57b006c560 0x7f57b006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.273+0000 7f57c37fe640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f57b006c560 0x7f57b006ea20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57b8000f60 tx=0x7f57b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.277+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f57bc032e20 con 0x7f57c415e920 2024-03-29T04:09:30.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.459+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f57c40d1a30 con 0x7f57c415e920 2024-03-29T04:09:30.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.459+0000 7f57abfff640 1 -- 172.21.15.156:0/3811357641 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f57bc07c760 con 0x7f57c415e920 2024-03-29T04:09:30.442 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f57b006c560 msgr2=0x7f57b006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f57b006c560 0x7f57b006ea20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57b8000f60 tx=0x7f57b8009040 comp rx=0 tx=0).stop 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 msgr2=0x7f57c40b69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 0x7f57c40b69d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009d90 tx=0x7f57bc004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 shutdown_connections 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f57b006c560 0x7f57b006ea20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57c415e920 0x7f57c40b69d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57c40d2120 0x7f57c40b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 --2- 172.21.15.156:0/3811357641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57c406cb10 0x7f57c40b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 >> 172.21.15.156:0/3811357641 conn(0x7f57c409d7a0 msgr2=0x7f57c40b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:30.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 shutdown_connections 2024-03-29T04:09:30.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.461+0000 7f57ca04a640 1 -- 172.21.15.156:0/3811357641 wait complete. 2024-03-29T04:09:30.452 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:30.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.592+0000 7fdce9baf640 1 Processor -- start 2024-03-29T04:09:30.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce9baf640 1 -- start start 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce9baf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdce40d2120 0x7fdce406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce9baf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce40d2df0 con 0x7fdce415e920 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce9baf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce40cf190 con 0x7fdce406cb10 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce9baf640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdce40cf7a0 con 0x7fdce40d2120 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdcdaffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdce40d2120 0x7fdce406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce37fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fdce415e920 0x7fdce40d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58082/0 (socket says 172.21.15.156:58082) 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.593+0000 7fdce37fe640 1 -- 172.21.15.156:0/2076338205 learned_addr learned my addr 172.21.15.156:0/2076338205 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.594+0000 7fdcdaffd640 1 -- 172.21.15.156:0/2076338205 >> v1:172.21.15.203:6789/0 conn(0x7fdce406cb10 legacy=0x7fdce40cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:30.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.594+0000 7fdcdaffd640 1 -- 172.21.15.156:0/2076338205 >> v1:172.21.15.156:6789/0 conn(0x7fdce415e920 legacy=0x7fdce40d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.594+0000 7fdcdaffd640 1 -- 172.21.15.156:0/2076338205 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce40d3080 con 0x7fdce40d2120 2024-03-29T04:09:30.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdcdaffd640 1 --2- 172.21.15.156:0/2076338205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce40d2120 0x7fdce406c5d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0097a0 tx=0x7fdccc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c4b952cdb672a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdce2ffd640 1 -- 172.21.15.156:0/2076338205 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc035020 con 0x7fdce40d2120 2024-03-29T04:09:30.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdce2ffd640 1 -- 172.21.15.156:0/2076338205 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdccc031830 con 0x7fdce40d2120 2024-03-29T04:09:30.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdce2ffd640 1 -- 172.21.15.156:0/2076338205 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc031c00 con 0x7fdce40d2120 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdce9baf640 1 -- 172.21.15.156:0/2076338205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce40d2120 msgr2=0x7fdce406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdce9baf640 1 --2- 172.21.15.156:0/2076338205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce40d2120 0x7fdce406c5d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0097a0 tx=0x7fdccc02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.595+0000 7fdce9baf640 1 -- 172.21.15.156:0/2076338205 shutdown_connections 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.596+0000 7fdce9baf640 1 --2- 172.21.15.156:0/2076338205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce40d2120 0x7fdce406c5d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.596+0000 7fdce9baf640 1 -- 172.21.15.156:0/2076338205 >> 172.21.15.156:0/2076338205 conn(0x7fdce409d7a0 msgr2=0x7fdce40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.596+0000 7fdce9baf640 1 -- 172.21.15.156:0/2076338205 shutdown_connections 2024-03-29T04:09:30.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.596+0000 7fdce9baf640 1 -- 172.21.15.156:0/2076338205 wait complete. 2024-03-29T04:09:30.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.597+0000 7fdce9baf640 1 Processor -- start 2024-03-29T04:09:30.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 -- start start 2024-03-29T04:09:30.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 0x7fdce4150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdce40d2120 0x7fdce41ea300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdce415e920 0x7fdce41ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdce40d34e0 con 0x7fdce406cb10 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdce40d3360 con 0x7fdce415e920 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce9baf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdce40d0340 con 0x7fdce40d2120 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 0x7fdce4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 0x7fdce4150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42958/0 (socket says 172.21.15.156:42958) 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.598+0000 7fdce37fe640 1 -- 172.21.15.156:0/1710271956 learned_addr learned my addr 172.21.15.156:0/1710271956 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdcdaffd640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdce40d2120 0x7fdce41ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce3fff640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdce415e920 0x7fdce41ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce37fe640 1 -- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdce40d2120 msgr2=0x7fdce41ea300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce37fe640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdce40d2120 0x7fdce41ea300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce37fe640 1 -- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdce415e920 msgr2=0x7fdce41ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce37fe640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdce415e920 0x7fdce41ea840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce37fe640 1 -- 172.21.15.156:0/1710271956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce415f920 con 0x7fdce406cb10 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce3fff640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdce415e920 0x7fdce41ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdcdaffd640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdce40d2120 0x7fdce41ea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce37fe640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 0x7fdce4150780 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400c400 tx=0x7fdcd400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4017070 con 0x7fdce406cb10 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd4005ce0 con 0x7fdce406cb10 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce415fc00 con 0x7fdce406cb10 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40072f0 con 0x7fdce406cb10 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.599+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce41601c0 con 0x7fdce406cb10 2024-03-29T04:09:30.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.600+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce40a2480 con 0x7fdce406cb10 2024-03-29T04:09:30.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.601+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdcd401a070 con 0x7fdce406cb10 2024-03-29T04:09:30.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.601+0000 7fdce0ff9640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc806c530 0x7fdcc806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.601+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdcd408ab90 con 0x7fdce406cb10 2024-03-29T04:09:30.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.602+0000 7fdcdaffd640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc806c530 0x7fdcc806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.602+0000 7fdcdaffd640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc806c530 0x7fdcc806e9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdccc036000 tx=0x7fdccc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.610+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdcd4059110 con 0x7fdce406cb10 2024-03-29T04:09:30.781 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.802+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdce404f9d0 con 0x7fdce406cb10 2024-03-29T04:09:30.781 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.802+0000 7fdce0ff9640 1 -- 172.21.15.156:0/1710271956 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fdcd4058a10 con 0x7fdce406cb10 2024-03-29T04:09:30.781 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc806c530 msgr2=0x7fdcc806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc806c530 0x7fdcc806e9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdccc036000 tx=0x7fdccc03c040 comp rx=0 tx=0).stop 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 msgr2=0x7fdce4150780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 0x7fdce4150780 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400c400 tx=0x7fdcd400c8c0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 shutdown_connections 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdcc806c530 0x7fdcc806e9f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdce415e920 0x7fdce41ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdce40d2120 0x7fdce41ea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 --2- 172.21.15.156:0/1710271956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce406cb10 0x7fdce4150780 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 >> 172.21.15.156:0/1710271956 conn(0x7fdce409d7a0 msgr2=0x7fdce4162900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 shutdown_connections 2024-03-29T04:09:30.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.805+0000 7fdce9baf640 1 -- 172.21.15.156:0/1710271956 wait complete. 2024-03-29T04:09:30.795 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:09:30.796 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:30.816 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:30.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.939+0000 7f790338d640 1 Processor -- start 2024-03-29T04:09:30.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.939+0000 7f790338d640 1 -- start start 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.939+0000 7f790338d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc1568f0 0x7f78fc0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.939+0000 7f790338d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f78fc0cc870 0x7f78fc0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790338d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78fc15cb50 con 0x7f78fc1609b0 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790338d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f78fc0cf8b0 con 0x7f78fc0cc870 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790338d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f78fc0cd200 con 0x7f78fc1568f0 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790192a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f78fc0cc870 0x7f78fc0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f7900928640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f78fc1609b0 0x7f78fc15d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58092/0 (socket says 172.21.15.156:58092) 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f7900928640 1 -- 172.21.15.156:0/3751184935 learned_addr learned my addr 172.21.15.156:0/3751184935 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f7901129640 1 --2- 172.21.15.156:0/3751184935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc1568f0 0x7f78fc0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790192a640 1 -- 172.21.15.156:0/3751184935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc1568f0 msgr2=0x7f78fc0cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790192a640 1 --2- 172.21.15.156:0/3751184935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc1568f0 0x7f78fc0cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790192a640 1 -- 172.21.15.156:0/3751184935 >> v1:172.21.15.156:6789/0 conn(0x7f78fc1609b0 legacy=0x7f78fc15d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f790192a640 1 -- 172.21.15.156:0/3751184935 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78fc160e30 con 0x7f78fc0cc870 2024-03-29T04:09:30.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.940+0000 7f78f3fff640 1 -- 172.21.15.156:0/3751184935 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887496026 0 0) 0x7f78fc15cb50 con 0x7f78fc1609b0 2024-03-29T04:09:30.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790192a640 1 --2- 172.21.15.156:0/3751184935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc0cc870 0x7f78fc0cccc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009d90 tx=0x7f78f402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85573b42ae8c7d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f78f3fff640 1 -- 172.21.15.156:0/3751184935 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4039070 con 0x7f78fc0cc870 2024-03-29T04:09:30.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f78f3fff640 1 -- 172.21.15.156:0/3751184935 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f402cbc0 con 0x7f78fc0cc870 2024-03-29T04:09:30.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f78f3fff640 1 -- 172.21.15.156:0/3751184935 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4034440 con 0x7f78fc0cc870 2024-03-29T04:09:30.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790338d640 1 -- 172.21.15.156:0/3751184935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc0cc870 msgr2=0x7f78fc0cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790338d640 1 --2- 172.21.15.156:0/3751184935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc0cc870 0x7f78fc0cccc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009d90 tx=0x7f78f402be10 comp rx=0 tx=0).stop 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790338d640 1 -- 172.21.15.156:0/3751184935 shutdown_connections 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790338d640 1 --2- 172.21.15.156:0/3751184935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc0cc870 0x7f78fc0cccc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790338d640 1 --2- 172.21.15.156:0/3751184935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc1568f0 0x7f78fc0cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.941+0000 7f790338d640 1 -- 172.21.15.156:0/3751184935 >> 172.21.15.156:0/3751184935 conn(0x7f78fc0b3eb0 msgr2=0x7f78fc0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.942+0000 7f790338d640 1 -- 172.21.15.156:0/3751184935 shutdown_connections 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.942+0000 7f790338d640 1 -- 172.21.15.156:0/3751184935 wait complete. 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 Processor -- start 2024-03-29T04:09:30.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 -- start start 2024-03-29T04:09:30.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc0cc870 0x7f78fc0b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78fc1568f0 0x7f78fc1efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 0x7f78fc1f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f78fc15cdf0 con 0x7f78fc1609b0 2024-03-29T04:09:30.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f78fc0cf5d0 con 0x7f78fc1568f0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.943+0000 7f790338d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f78fc156cd0 con 0x7f78fc0cc870 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 0x7f78fc1f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 0x7f78fc1f46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42976/0 (socket says 172.21.15.156:42976) 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 -- 172.21.15.156:0/240885894 learned_addr learned my addr 172.21.15.156:0/240885894 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f7901129640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc0cc870 0x7f78fc0b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f7900928640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78fc1568f0 0x7f78fc1efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 -- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc0cc870 msgr2=0x7f78fc0b73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc0cc870 0x7f78fc0b73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 -- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78fc1568f0 msgr2=0x7f78fc1efac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78fc1568f0 0x7f78fc1efac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 -- 172.21.15.156:0/240885894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78fc0b78f0 con 0x7f78fc1609b0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790192a640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 0x7f78fc1f46a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009d90 tx=0x7f78f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4039050 con 0x7f78fc1609b0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f4045070 con 0x7f78fc1609b0 2024-03-29T04:09:30.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78fc1f3600 con 0x7f78fc1609b0 2024-03-29T04:09:30.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.944+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f403e830 con 0x7f78fc1609b0 2024-03-29T04:09:30.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.945+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78fc1f3bc0 con 0x7f78fc1609b0 2024-03-29T04:09:30.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.946+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f78f403e9d0 con 0x7f78fc1609b0 2024-03-29T04:09:30.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.946+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78fc068610 con 0x7f78fc1609b0 2024-03-29T04:09:30.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.946+0000 7f78f1ffb640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f78e006c530 0x7f78e006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:30.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.947+0000 7f7901129640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f78e006c530 0x7f78e006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:30.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.947+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f78f40ae430 con 0x7f78fc1609b0 2024-03-29T04:09:30.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.947+0000 7f7901129640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f78e006c530 0x7f78e006e9f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f78f8006fd0 tx=0x7f78f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:30.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:30.951+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f78f407c530 con 0x7f78fc1609b0 2024-03-29T04:09:31.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.215+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78fc15dc30 con 0x7f78fc1609b0 2024-03-29T04:09:31.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.216+0000 7f78f1ffb640 1 -- 172.21.15.156:0/240885894 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f78f4080830 con 0x7f78fc1609b0 2024-03-29T04:09:31.196 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:31.196 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f78e006c530 msgr2=0x7f78e006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f78e006c530 0x7f78e006e9f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f78f8006fd0 tx=0x7f78f8008040 comp rx=0 tx=0).stop 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 msgr2=0x7f78fc1f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 0x7f78fc1f46a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009d90 tx=0x7f78f4004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 shutdown_connections 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f78e006c530 0x7f78e006e9f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78fc1609b0 0x7f78fc1f46a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78fc1568f0 0x7f78fc1efac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 --2- 172.21.15.156:0/240885894 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78fc0cc870 0x7f78fc0b73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 >> 172.21.15.156:0/240885894 conn(0x7f78fc0b3eb0 msgr2=0x7f78fc13e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:31.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 shutdown_connections 2024-03-29T04:09:31.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.219+0000 7f790338d640 1 -- 172.21.15.156:0/240885894 wait complete. 2024-03-29T04:09:31.209 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:31.210 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:31.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.338+0000 7fe804cb2640 1 Processor -- start 2024-03-29T04:09:31.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.338+0000 7fe804cb2640 1 -- start start 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe804cb2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe80015e920 0x7fe8000d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe804cb2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000cfc30 0x7fe8000d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe804cb2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8000cf190 con 0x7fe8000d2120 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe804cb2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe8000cf7a0 con 0x7fe80015e920 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe804cb2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe8000d3080 con 0x7fe8000cfc30 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe7fe59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe80015e920 0x7fe8000d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe7f5d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fe8000d2120 0x7fe80006c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:58106/0 (socket says 172.21.15.156:58106) 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.339+0000 7fe7f5d9b640 1 -- 172.21.15.156:0/2001157087 learned_addr learned my addr 172.21.15.156:0/2001157087 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:31.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2001157087 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000cfc30 msgr2=0x7fe8000d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fed9d640 1 --2- 172.21.15.156:0/2001157087 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000cfc30 0x7fe8000d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fe59c640 1 --2- 172.21.15.156:0/2001157087 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000cfc30 0x7fe8000d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2001157087 >> v1:172.21.15.156:6789/0 conn(0x7fe8000d2120 legacy=0x7fe80006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2001157087 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8000cf430 con 0x7fe80015e920 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fdd9b640 1 -- 172.21.15.156:0/2001157087 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 885026785 0 0) 0x7fe8000cf190 con 0x7fe8000d2120 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.340+0000 7fe7fe59c640 1 --2- 172.21.15.156:0/2001157087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe80015e920 0x7fe8000d26e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00097a0 tx=0x7fe7f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff9b4a1259f5b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe7fdd9b640 1 -- 172.21.15.156:0/2001157087 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0035020 con 0x7fe80015e920 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe7fdd9b640 1 -- 172.21.15.156:0/2001157087 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f0031830 con 0x7fe80015e920 2024-03-29T04:09:31.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe7fdd9b640 1 -- 172.21.15.156:0/2001157087 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0031c00 con 0x7fe80015e920 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 -- 172.21.15.156:0/2001157087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe80015e920 msgr2=0x7fe8000d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2001157087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe80015e920 0x7fe8000d26e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00097a0 tx=0x7fe7f002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 -- 172.21.15.156:0/2001157087 shutdown_connections 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2001157087 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000cfc30 0x7fe8000d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2001157087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe80015e920 0x7fe8000d26e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 -- 172.21.15.156:0/2001157087 >> 172.21.15.156:0/2001157087 conn(0x7fe80009d7a0 msgr2=0x7fe8000d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 -- 172.21.15.156:0/2001157087 shutdown_connections 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.341+0000 7fe804cb2640 1 -- 172.21.15.156:0/2001157087 wait complete. 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 Processor -- start 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 -- start start 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 0x7fe80014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000d2120 0x7fe8000cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe80015e920 0x7fe8000cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000d34e0 con 0x7fe8000cfc30 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000d3360 con 0x7fe80015e920 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe804cb2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe8000d2c20 con 0x7fe8000d2120 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe7fe59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 0x7fe80014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe7fe59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 0x7fe80014d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42984/0 (socket says 172.21.15.156:42984) 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.343+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2044140740 learned_addr learned my addr 172.21.15.156:0/2044140740 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fed9d640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe80015e920 0x7fe8000cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7f5d9b640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000d2120 0x7fe8000cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000d2120 msgr2=0x7fe8000cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fe59c640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000d2120 0x7fe8000cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe80015e920 msgr2=0x7fe8000cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fe59c640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe80015e920 0x7fe8000cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fe59c640 1 -- 172.21.15.156:0/2044140740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8000ccbe0 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fed9d640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe80015e920 0x7fe8000cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7f5d9b640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000d2120 0x7fe8000cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7fe59c640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 0x7fe80014d8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0034040 tx=0x7fe7f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0040070 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f0039a20 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8000cce70 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0035040 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.344+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8000cd430 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.345+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8000a2480 con 0x7fe8000cfc30 2024-03-29T04:09:31.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.346+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe7f0038020 con 0x7fe8000cfc30 2024-03-29T04:09:31.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.346+0000 7fe7f77fe640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe7e006c530 0x7fe7e006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.346+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe7f00ad230 con 0x7fe8000cfc30 2024-03-29T04:09:31.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.346+0000 7fe7f5d9b640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe7e006c530 0x7fe7e006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.347+0000 7fe7f5d9b640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe7e006c530 0x7fe7e006e9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe8000caab0 tx=0x7fe7e8007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.350+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe7f007b7c0 con 0x7fe8000cfc30 2024-03-29T04:09:31.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.530+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe80004f9d0 con 0x7fe8000cfc30 2024-03-29T04:09:31.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.531+0000 7fe7f77fe640 1 -- 172.21.15.156:0/2044140740 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fe7f007b0c0 con 0x7fe8000cfc30 2024-03-29T04:09:31.511 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:31.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe7e006c530 msgr2=0x7fe7e006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe7e006c530 0x7fe7e006e9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe8000caab0 tx=0x7fe7e8007c10 comp rx=0 tx=0).stop 2024-03-29T04:09:31.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 msgr2=0x7fe80014d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 0x7fe80014d8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0034040 tx=0x7fe7f00046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 shutdown_connections 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe7e006c530 0x7fe7e006e9f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe80015e920 0x7fe8000cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8000d2120 0x7fe8000cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 --2- 172.21.15.156:0/2044140740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8000cfc30 0x7fe80014d8f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 >> 172.21.15.156:0/2044140740 conn(0x7fe80009d7a0 msgr2=0x7fe8000d4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 shutdown_connections 2024-03-29T04:09:31.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.533+0000 7fe804cb2640 1 -- 172.21.15.156:0/2044140740 wait complete. 2024-03-29T04:09:31.524 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:31.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.649+0000 7f9167874640 1 Processor -- start 2024-03-29T04:09:31.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.650+0000 7f9167874640 1 -- start start 2024-03-29T04:09:31.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.650+0000 7f9167874640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f916015e920 0x7f91600d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.650+0000 7f9167874640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f91600d2120 0x7f916006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.650+0000 7f9167874640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91600cf190 con 0x7f916006cb10 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.650+0000 7f9167874640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f91600cf7a0 con 0x7f916015e920 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.650+0000 7f9167874640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f91600d3080 con 0x7f91600d2120 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f916015e920 0x7f91600d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f916015e920 0x7f91600d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43544/0 (socket says 172.21.15.156:43544) 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 -- 172.21.15.156:0/1574201859 learned_addr learned my addr 172.21.15.156:0/1574201859 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9164e0f640 1 --2- 172.21.15.156:0/1574201859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f91600d2120 0x7f916006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 -- 172.21.15.156:0/1574201859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f91600d2120 msgr2=0x7f916006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 --2- 172.21.15.156:0/1574201859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f91600d2120 0x7f916006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 -- 172.21.15.156:0/1574201859 >> v1:172.21.15.203:6789/0 conn(0x7f916006cb10 legacy=0x7f91600cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.651+0000 7f9165610640 1 -- 172.21.15.156:0/1574201859 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91600cf430 con 0x7f916015e920 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9165610640 1 --2- 172.21.15.156:0/1574201859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f916015e920 0x7f91600d26e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443872611ad9ff9d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9157fff640 1 -- 172.21.15.156:0/1574201859 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c034070 con 0x7f916015e920 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9157fff640 1 -- 172.21.15.156:0/1574201859 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f915c030840 con 0x7f916015e920 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9157fff640 1 -- 172.21.15.156:0/1574201859 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c030c10 con 0x7f916015e920 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9167874640 1 -- 172.21.15.156:0/1574201859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f916015e920 msgr2=0x7f91600d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9167874640 1 --2- 172.21.15.156:0/1574201859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f916015e920 0x7f91600d26e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c02b980 comp rx=0 tx=0).stop 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9167874640 1 -- 172.21.15.156:0/1574201859 shutdown_connections 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9167874640 1 --2- 172.21.15.156:0/1574201859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f91600d2120 0x7f916006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9167874640 1 --2- 172.21.15.156:0/1574201859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f916015e920 0x7f91600d26e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.652+0000 7f9167874640 1 -- 172.21.15.156:0/1574201859 >> 172.21.15.156:0/1574201859 conn(0x7f916009d7a0 msgr2=0x7f91600d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.653+0000 7f9167874640 1 -- 172.21.15.156:0/1574201859 shutdown_connections 2024-03-29T04:09:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.653+0000 7f9167874640 1 -- 172.21.15.156:0/1574201859 wait complete. 2024-03-29T04:09:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.654+0000 7f9167874640 1 Processor -- start 2024-03-29T04:09:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.654+0000 7f9167874640 1 -- start start 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9167874640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f916006cb10 0x7f91600cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9167874640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 0x7f9160160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9167874640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f916015e920 0x7f91600c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9167874640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91600d04c0 con 0x7f91600d2120 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9167874640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91600d0340 con 0x7f916006cb10 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9167874640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f91600d2c20 con 0x7f916015e920 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9164e0f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 0x7f9160160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9164e0f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 0x7f9160160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43550/0 (socket says 172.21.15.156:43550) 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.655+0000 7f9164e0f640 1 -- 172.21.15.156:0/673584117 learned_addr learned my addr 172.21.15.156:0/673584117 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9165610640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f916006cb10 0x7f91600cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9165e11640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f916015e920 0x7f91600c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9164e0f640 1 -- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f916015e920 msgr2=0x7f91600c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9164e0f640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f916015e920 0x7f91600c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9164e0f640 1 -- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f916006cb10 msgr2=0x7f91600cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9164e0f640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f916006cb10 0x7f91600cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9164e0f640 1 -- 172.21.15.156:0/673584117 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91600ccac0 con 0x7f91600d2120 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9165610640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f916006cb10 0x7f91600cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9165e11640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f916015e920 0x7f91600c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9164e0f640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 0x7f9160160560 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f915000d800 tx=0x7f915000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915000b6c0 con 0x7f91600d2120 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9150019070 con 0x7f91600d2120 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91600cce00 con 0x7f91600d2120 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915000ba80 con 0x7f91600d2120 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.656+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91600cd640 con 0x7f91600d2120 2024-03-29T04:09:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.657+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91600a2480 con 0x7f91600d2120 2024-03-29T04:09:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.658+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f915000bc20 con 0x7f91600d2120 2024-03-29T04:09:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.658+0000 7f9155ffb640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f914406c530 0x7f914406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.658+0000 7f9165610640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f914406c530 0x7f914406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.658+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f915008bba0 con 0x7f91600d2120 2024-03-29T04:09:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.659+0000 7f9165610640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f914406c530 0x7f914406e9f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f915c036000 tx=0x7f915c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.663+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f91600cce00 con 0x7f91600d2120 2024-03-29T04:09:31.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.843+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f916004f9d0 con 0x7f91600d2120 2024-03-29T04:09:31.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.843+0000 7f9155ffb640 1 -- 172.21.15.156:0/673584117 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f915005a1c0 con 0x7f91600d2120 2024-03-29T04:09:31.823 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:31.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f914406c530 msgr2=0x7f914406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f914406c530 0x7f914406e9f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f915c036000 tx=0x7f915c03b040 comp rx=0 tx=0).stop 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 msgr2=0x7f9160160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 0x7f9160160560 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f915000d800 tx=0x7f915000dcc0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 shutdown_connections 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f914406c530 0x7f914406e9f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f916015e920 0x7f91600c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91600d2120 0x7f9160160560 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 --2- 172.21.15.156:0/673584117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f916006cb10 0x7f91600cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 >> 172.21.15.156:0/673584117 conn(0x7f916009d7a0 msgr2=0x7f91600b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 shutdown_connections 2024-03-29T04:09:31.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.846+0000 7f9167874640 1 -- 172.21.15.156:0/673584117 wait complete. 2024-03-29T04:09:31.836 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:31.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.984+0000 7f03c6b46640 1 Processor -- start 2024-03-29T04:09:31.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.984+0000 7f03c6b46640 1 -- start start 2024-03-29T04:09:31.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03c6b46640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c00af6b0 0x7f03c00b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03c6b46640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f03c015c990 0x7f03c015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03c6b46640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c00b3050 con 0x7f03c00d0280 2024-03-29T04:09:31.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03c6b46640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f03c00afa90 con 0x7f03c015c990 2024-03-29T04:09:31.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03c6b46640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f03c015d510 con 0x7f03c00af6b0 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f03c015c990 0x7f03c015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f03c015c990 0x7f03c015ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43556/0 (socket says 172.21.15.156:43556) 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 -- 172.21.15.156:0/3542399447 learned_addr learned my addr 172.21.15.156:0/3542399447 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03c48e2640 1 --2- 172.21.15.156:0/3542399447 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c00af6b0 0x7f03c00b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 -- 172.21.15.156:0/3542399447 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c00af6b0 msgr2=0x7f03c00b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 --2- 172.21.15.156:0/3542399447 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c00af6b0 0x7f03c00b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 -- 172.21.15.156:0/3542399447 >> v1:172.21.15.203:6789/0 conn(0x7f03c00d0280 legacy=0x7f03c015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.985+0000 7f03bffff640 1 -- 172.21.15.156:0/3542399447 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c015d7f0 con 0x7f03c015c990 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.986+0000 7f03bffff640 1 --2- 172.21.15.156:0/3542399447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c015c990 0x7f03c015ce10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009870 tx=0x7f03ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde1530930a3f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.986+0000 7f03bf7fe640 1 -- 172.21.15.156:0/3542399447 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac034070 con 0x7f03c015c990 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.986+0000 7f03bf7fe640 1 -- 172.21.15.156:0/3542399447 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ac030840 con 0x7f03c015c990 2024-03-29T04:09:31.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.986+0000 7f03bf7fe640 1 -- 172.21.15.156:0/3542399447 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac030c10 con 0x7f03c015c990 2024-03-29T04:09:31.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 -- 172.21.15.156:0/3542399447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c015c990 msgr2=0x7f03c015ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3542399447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c015c990 0x7f03c015ce10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009870 tx=0x7f03ac02b980 comp rx=0 tx=0).stop 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 -- 172.21.15.156:0/3542399447 shutdown_connections 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3542399447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c015c990 0x7f03c015ce10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3542399447 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c00af6b0 0x7f03c00b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 -- 172.21.15.156:0/3542399447 >> 172.21.15.156:0/3542399447 conn(0x7f03c00b1f30 msgr2=0x7f03c00b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 -- 172.21.15.156:0/3542399447 shutdown_connections 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.987+0000 7f03c6b46640 1 -- 172.21.15.156:0/3542399447 wait complete. 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.988+0000 7f03c6b46640 1 Processor -- start 2024-03-29T04:09:31.968 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.988+0000 7f03c6b46640 1 -- start start 2024-03-29T04:09:31.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c6b46640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03c00af6b0 0x7f03c00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c6b46640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 0x7f03c00ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c6b46640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c015c990 0x7f03c014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c6b46640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f03c00ad200 con 0x7f03c00d0280 2024-03-29T04:09:31.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c6b46640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03c00b35f0 con 0x7f03c00af6b0 2024-03-29T04:09:31.970 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c6b46640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f03c00d0710 con 0x7f03c015c990 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 0x7f03c00ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 0x7f03c00ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43570/0 (socket says 172.21.15.156:43570) 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 -- 172.21.15.156:0/3111918453 learned_addr learned my addr 172.21.15.156:0/3111918453 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c50e3640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c015c990 0x7f03c014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c48e2640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03c00af6b0 0x7f03c00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 -- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c015c990 msgr2=0x7f03c014f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c015c990 0x7f03c014f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 -- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03c00af6b0 msgr2=0x7f03c00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03c00af6b0 0x7f03c00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03bffff640 1 -- 172.21.15.156:0/3111918453 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c014fe30 con 0x7f03c00d0280 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c50e3640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c015c990 0x7f03c014f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.989+0000 7f03c48e2640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03c00af6b0 0x7f03c00cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.990+0000 7f03bffff640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 0x7f03c00ce140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0090b0 tx=0x7f03ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.990+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac034040 con 0x7f03c00d0280 2024-03-29T04:09:31.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.990+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c00cd0a0 con 0x7f03c00d0280 2024-03-29T04:09:31.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.990+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ac03ada0 con 0x7f03c00d0280 2024-03-29T04:09:31.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.990+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac044610 con 0x7f03c00d0280 2024-03-29T04:09:31.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.990+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c00cd660 con 0x7f03c00d0280 2024-03-29T04:09:31.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.991+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c009e3b0 con 0x7f03c00d0280 2024-03-29T04:09:31.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.991+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f03ac04b060 con 0x7f03c00d0280 2024-03-29T04:09:31.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.991+0000 7f03bd7fa640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f03a406c530 0x7f03a406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:31.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.992+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f03ac0ad750 con 0x7f03c00d0280 2024-03-29T04:09:31.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.992+0000 7f03c48e2640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f03a406c530 0x7f03a406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:31.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.992+0000 7f03c48e2640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f03a406c530 0x7f03a406e9f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f03b4006fd0 tx=0x7f03b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:31.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:31.996+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f03ac03f070 con 0x7f03c00d0280 2024-03-29T04:09:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.205+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f03c004f9d0 con 0x7f03c00d0280 2024-03-29T04:09:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.205+0000 7f03bd7fa640 1 -- 172.21.15.156:0/3111918453 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f03ac07bd80 con 0x7f03c00d0280 2024-03-29T04:09:32.185 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f03a406c530 msgr2=0x7f03a406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f03a406c530 0x7f03a406e9f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f03b4006fd0 tx=0x7f03b4008040 comp rx=0 tx=0).stop 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 msgr2=0x7f03c00ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 0x7f03c00ce140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0090b0 tx=0x7f03ac0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 shutdown_connections 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f03a406c530 0x7f03a406e9f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03c015c990 0x7f03c014f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c00d0280 0x7f03c00ce140 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 --2- 172.21.15.156:0/3111918453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03c00af6b0 0x7f03c00cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 >> 172.21.15.156:0/3111918453 conn(0x7f03c00b1f30 msgr2=0x7f03c01529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 shutdown_connections 2024-03-29T04:09:32.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.208+0000 7f03c6b46640 1 -- 172.21.15.156:0/3111918453 wait complete. 2024-03-29T04:09:32.199 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:32.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.324+0000 7f853f67e640 1 Processor -- start 2024-03-29T04:09:32.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.324+0000 7f853f67e640 1 -- start start 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853f67e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f85380d2120 0x7f853806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853f67e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f853806cb10 0x7f85380cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853f67e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85380d2df0 con 0x7f853815e920 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853f67e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f85380cf7a0 con 0x7f85380d2120 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853f67e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f85380cf190 con 0x7f853806cb10 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853cc19640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f85380d2120 0x7f853806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853d41a640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f853815e920 0x7f85380d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33204/0 (socket says 172.21.15.156:33204) 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.325+0000 7f853d41a640 1 -- 172.21.15.156:0/1765278371 learned_addr learned my addr 172.21.15.156:0/1765278371 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:32.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.326+0000 7f853dc1b640 1 --2- 172.21.15.156:0/1765278371 >> v2:172.21.15.203:3300/0 conn(0x7f853806cb10 0x7f85380cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.326+0000 7f853cc19640 1 -- 172.21.15.156:0/1765278371 >> v2:172.21.15.203:3300/0 conn(0x7f853806cb10 msgr2=0x7f85380cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:32.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.326+0000 7f853cc19640 1 --2- 172.21.15.156:0/1765278371 >> v2:172.21.15.203:3300/0 conn(0x7f853806cb10 0x7f85380cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.326+0000 7f853cc19640 1 -- 172.21.15.156:0/1765278371 >> v1:172.21.15.156:6789/0 conn(0x7f853815e920 legacy=0x7f85380d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.326+0000 7f852ffff640 1 -- 172.21.15.156:0/1765278371 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 376666048 0 0) 0x7f85380d2df0 con 0x7f853815e920 2024-03-29T04:09:32.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.326+0000 7f853cc19640 1 -- 172.21.15.156:0/1765278371 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85380d3080 con 0x7f85380d2120 2024-03-29T04:09:32.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.327+0000 7f853cc19640 1 --2- 172.21.15.156:0/1765278371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85380d2120 0x7f853806c5d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f853808ca00 tx=0x7f852802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781891ba49ae52d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:32.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.327+0000 7f852ffff640 1 -- 172.21.15.156:0/1765278371 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528035020 con 0x7f85380d2120 2024-03-29T04:09:32.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.327+0000 7f852ffff640 1 -- 172.21.15.156:0/1765278371 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8528031830 con 0x7f85380d2120 2024-03-29T04:09:32.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.327+0000 7f852ffff640 1 -- 172.21.15.156:0/1765278371 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528031c00 con 0x7f85380d2120 2024-03-29T04:09:32.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 -- 172.21.15.156:0/1765278371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85380d2120 msgr2=0x7f853806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 --2- 172.21.15.156:0/1765278371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85380d2120 0x7f853806c5d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f853808ca00 tx=0x7f852802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 -- 172.21.15.156:0/1765278371 shutdown_connections 2024-03-29T04:09:32.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 --2- 172.21.15.156:0/1765278371 >> v2:172.21.15.203:3300/0 conn(0x7f853806cb10 0x7f85380cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 --2- 172.21.15.156:0/1765278371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85380d2120 0x7f853806c5d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 -- 172.21.15.156:0/1765278371 >> 172.21.15.156:0/1765278371 conn(0x7f853809d7a0 msgr2=0x7f85380d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:32.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 -- 172.21.15.156:0/1765278371 shutdown_connections 2024-03-29T04:09:32.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.328+0000 7f853f67e640 1 -- 172.21.15.156:0/1765278371 wait complete. 2024-03-29T04:09:32.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.330+0000 7f853f67e640 1 Processor -- start 2024-03-29T04:09:32.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.330+0000 7f853f67e640 1 -- start start 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853f67e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 0x7f853814e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853f67e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85380d2120 0x7f85380b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853f67e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f853815e920 0x7f85380b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853f67e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85380d34e0 con 0x7f853806cb10 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853f67e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85380d3360 con 0x7f853815e920 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853f67e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f85380d0170 con 0x7f85380d2120 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 0x7f853814e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 0x7f853814e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43588/0 (socket says 172.21.15.156:43588) 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 -- 172.21.15.156:0/1403277419 learned_addr learned my addr 172.21.15.156:0/1403277419 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 -- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85380d2120 msgr2=0x7f85380b7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853cc19640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85380d2120 0x7f85380b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853dc1b640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f853815e920 0x7f85380b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85380d2120 0x7f85380b7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 -- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f853815e920 msgr2=0x7f85380b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f853815e920 0x7f85380b75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 -- 172.21.15.156:0/1403277419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85380b9ff0 con 0x7f853806cb10 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.331+0000 7f853d41a640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 0x7f853814e440 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f853400c430 tx=0x7f853400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.332+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340074e0 con 0x7f853806cb10 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.332+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85340047e0 con 0x7f853806cb10 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.332+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534004bd0 con 0x7f853806cb10 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.332+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85380b9d10 con 0x7f853806cb10 2024-03-29T04:09:32.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.332+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85380b49e0 con 0x7f853806cb10 2024-03-29T04:09:32.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.333+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85380a2480 con 0x7f853806cb10 2024-03-29T04:09:32.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.333+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8534011070 con 0x7f853806cb10 2024-03-29T04:09:32.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.334+0000 7f852dffb640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f851c06c530 0x7f851c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.334+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f853408b3b0 con 0x7f853806cb10 2024-03-29T04:09:32.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.334+0000 7f853cc19640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f851c06c530 0x7f851c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.334+0000 7f853cc19640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f851c06c530 0x7f851c06e9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8528036000 tx=0x7f8528008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:32.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.338+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f85340599d0 con 0x7f853806cb10 2024-03-29T04:09:32.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.545+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f853804f9d0 con 0x7f853806cb10 2024-03-29T04:09:32.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.545+0000 7f852dffb640 1 -- 172.21.15.156:0/1403277419 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f85340592d0 con 0x7f853806cb10 2024-03-29T04:09:32.525 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f851c06c530 msgr2=0x7f851c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f851c06c530 0x7f851c06e9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8528036000 tx=0x7f8528008030 comp rx=0 tx=0).stop 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 msgr2=0x7f853814e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 0x7f853814e440 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f853400c430 tx=0x7f853400c8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 shutdown_connections 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f851c06c530 0x7f851c06e9f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f853815e920 0x7f85380b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85380d2120 0x7f85380b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 --2- 172.21.15.156:0/1403277419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f853806cb10 0x7f853814e440 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 >> 172.21.15.156:0/1403277419 conn(0x7f853809d7a0 msgr2=0x7f85380d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:32.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 shutdown_connections 2024-03-29T04:09:32.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.548+0000 7f853f67e640 1 -- 172.21.15.156:0/1403277419 wait complete. 2024-03-29T04:09:32.538 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi203.front.sepia.ceph.com... 2024-03-29T04:09:32.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:32.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-29T04:09:32.572 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:32.572 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:09:32.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:32.572 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-29T04:09:32.700 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:09:32.700 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:32.717 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:09:32.717 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:32.773 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:32.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.936+0000 7fc0ed5f4640 1 Processor -- start 2024-03-29T04:09:32.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.936+0000 7fc0ed5f4640 1 -- start start 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0ed5f4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc0e80af6b0 0x7fc0e815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0ed5f4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc0e80b3050 0x7fc0e806f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0ed5f4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e815c7e0 con 0x7fc0e806f5d0 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0ed5f4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e8072760 con 0x7fc0e80af6b0 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0ed5f4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e80afa90 con 0x7fc0e80b3050 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e77fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc0e806f5d0 0x7fc0e815ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33212/0 (socket says 172.21.15.156:33212) 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e6ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc0e80af6b0 0x7fc0e815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e77fe640 1 -- 172.21.15.156:0/2053399809 learned_addr learned my addr 172.21.15.156:0/2053399809 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e67fc640 1 --2- 172.21.15.156:0/2053399809 >> v2:172.21.15.203:3300/0 conn(0x7fc0e80b3050 0x7fc0e806f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/2053399809 >> v2:172.21.15.203:3300/0 conn(0x7fc0e80b3050 msgr2=0x7fc0e806f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e6ffd640 1 --2- 172.21.15.156:0/2053399809 >> v2:172.21.15.203:3300/0 conn(0x7fc0e80b3050 0x7fc0e806f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/2053399809 >> v1:172.21.15.156:6789/0 conn(0x7fc0e806f5d0 legacy=0x7fc0e815ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.937+0000 7fc0e5ffb640 1 -- 172.21.15.156:0/2053399809 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187700329 0 0) 0x7fc0e0001110 con 0x7fc0e806f5d0 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.938+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/2053399809 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e80b3510 con 0x7fc0e80af6b0 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.938+0000 7fc0e67fc640 1 --2- 172.21.15.156:0/2053399809 >> v2:172.21.15.203:3300/0 conn(0x7fc0e80b3050 0x7fc0e806f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.938+0000 7fc0e6ffd640 1 --2- 172.21.15.156:0/2053399809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e80af6b0 0x7fc0e815c2a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8075030 tx=0x7fc0d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdd41fbb0ba7d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.938+0000 7fc0e5ffb640 1 -- 172.21.15.156:0/2053399809 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8039070 con 0x7fc0e80af6b0 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.938+0000 7fc0e5ffb640 1 -- 172.21.15.156:0/2053399809 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d802cbf0 con 0x7fc0e80af6b0 2024-03-29T04:09:32.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.938+0000 7fc0e5ffb640 1 -- 172.21.15.156:0/2053399809 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8034440 con 0x7fc0e80af6b0 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/2053399809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e80af6b0 msgr2=0x7fc0e815c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/2053399809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e80af6b0 0x7fc0e815c2a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8075030 tx=0x7fc0d802be40 comp rx=0 tx=0).stop 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/2053399809 shutdown_connections 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/2053399809 >> v2:172.21.15.203:3300/0 conn(0x7fc0e80b3050 0x7fc0e806f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/2053399809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e80af6b0 0x7fc0e815c2a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/2053399809 >> 172.21.15.156:0/2053399809 conn(0x7fc0e80b1d50 msgr2=0x7fc0e80b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/2053399809 shutdown_connections 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.939+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/2053399809 wait complete. 2024-03-29T04:09:32.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.940+0000 7fc0ed5f4640 1 Processor -- start 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.940+0000 7fc0ed5f4640 1 -- start start 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0ed5f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 0x7fc0e8144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0ed5f4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e80af6b0 0x7fc0e81475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0ed5f4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e80b3050 0x7fc0e81e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0ed5f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e80b3db0 con 0x7fc0e806f5d0 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0ed5f4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e80b3ad0 con 0x7fc0e80b3050 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0ed5f4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e80b4090 con 0x7fc0e80af6b0 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 0x7fc0e8144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 0x7fc0e8144420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43620/0 (socket says 172.21.15.156:43620) 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.941+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/1495057030 learned_addr learned my addr 172.21.15.156:0/1495057030 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e77fe640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e80b3050 0x7fc0e81e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e80af6b0 msgr2=0x7fc0e81475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e6ffd640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e80af6b0 0x7fc0e81475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e80b3050 msgr2=0x7fc0e81e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e6ffd640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e80b3050 0x7fc0e81e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e6ffd640 1 -- 172.21.15.156:0/1495057030 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e8159ec0 con 0x7fc0e806f5d0 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0e6ffd640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 0x7fc0e8144420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009e80 tx=0x7fc0d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8039050 con 0x7fc0e806f5d0 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d8045070 con 0x7fc0e806f5d0 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e8147b20 con 0x7fc0e806f5d0 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d803ea50 con 0x7fc0e806f5d0 2024-03-29T04:09:32.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.942+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e8156580 con 0x7fc0e806f5d0 2024-03-29T04:09:32.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.943+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e8144ba0 con 0x7fc0e806f5d0 2024-03-29T04:09:32.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.943+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc0d802ca70 con 0x7fc0e806f5d0 2024-03-29T04:09:32.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.944+0000 7fc0df7fe640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc0c806c530 0x7fc0c806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:32.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.944+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc0d80ae560 con 0x7fc0e806f5d0 2024-03-29T04:09:32.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.944+0000 7fc0e67fc640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc0c806c530 0x7fc0c806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:32.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.944+0000 7fc0e67fc640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc0c806c530 0x7fc0c806e9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0006fd0 tx=0x7fc0d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:32.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:32.948+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc0e8147b20 con 0x7fc0e806f5d0 2024-03-29T04:09:33.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.136+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0e8068610 con 0x7fc0e806f5d0 2024-03-29T04:09:33.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.137+0000 7fc0df7fe640 1 -- 172.21.15.156:0/1495057030 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fc0e8159ec0 con 0x7fc0e806f5d0 2024-03-29T04:09:33.117 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:33.117 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":21,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:08:54.808842+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc0c806c530 msgr2=0x7fc0c806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc0c806c530 0x7fc0c806e9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0006fd0 tx=0x7fc0d0008040 comp rx=0 tx=0).stop 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 msgr2=0x7fc0e8144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 0x7fc0e8144420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009e80 tx=0x7fc0d8004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 shutdown_connections 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc0c806c530 0x7fc0c806e9f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e80b3050 0x7fc0e81e5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e80af6b0 0x7fc0e81475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 --2- 172.21.15.156:0/1495057030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0e806f5d0 0x7fc0e8144420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 >> 172.21.15.156:0/1495057030 conn(0x7fc0e80b1d50 msgr2=0x7fc0e80afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:33.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 shutdown_connections 2024-03-29T04:09:33.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.139+0000 7fc0ed5f4640 1 -- 172.21.15.156:0/1495057030 wait complete. 2024-03-29T04:09:33.130 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:33.131 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:33.268 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.289+0000 7ffafd790640 1 Processor -- start 2024-03-29T04:09:33.269 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.289+0000 7ffafd790640 1 -- start start 2024-03-29T04:09:33.270 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffafd790640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffaf8156ac0 0x7ffaf80ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffafd790640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffaf806eaa0 0x7ffaf815d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffafd790640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaf8156ea0 con 0x7ffaf80b5df0 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffafd790640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ffaf80cf1f0 con 0x7ffaf806eaa0 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffafd790640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ffaf815cec0 con 0x7ffaf8156ac0 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffaf806eaa0 0x7ffaf815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffaf806eaa0 0x7ffaf815d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43624/0 (socket says 172.21.15.156:43624) 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 -- 172.21.15.156:0/672188257 learned_addr learned my addr 172.21.15.156:0/672188257 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaf6ffd640 1 --2- 172.21.15.156:0/672188257 >> v2:172.21.15.203:3300/0 conn(0x7ffaf8156ac0 0x7ffaf80ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 -- 172.21.15.156:0/672188257 >> v2:172.21.15.203:3300/0 conn(0x7ffaf8156ac0 msgr2=0x7ffaf80ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 --2- 172.21.15.156:0/672188257 >> v2:172.21.15.203:3300/0 conn(0x7ffaf8156ac0 0x7ffaf80ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 -- 172.21.15.156:0/672188257 >> v1:172.21.15.203:6789/0 conn(0x7ffaf80b5df0 legacy=0x7ffaf80b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.290+0000 7ffaee7fc640 1 -- 172.21.15.156:0/672188257 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf806ef50 con 0x7ffaf806eaa0 2024-03-29T04:09:33.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.291+0000 7ffaee7fc640 1 --2- 172.21.15.156:0/672188257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815d350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ffaf806ce70 tx=0x7ffae002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1438e2e5f48d9ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.291+0000 7ffaf67fc640 1 -- 172.21.15.156:0/672188257 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0035020 con 0x7ffaf806eaa0 2024-03-29T04:09:33.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.291+0000 7ffaf67fc640 1 -- 172.21.15.156:0/672188257 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae0031870 con 0x7ffaf806eaa0 2024-03-29T04:09:33.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.291+0000 7ffaf67fc640 1 -- 172.21.15.156:0/672188257 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0031c40 con 0x7ffaf806eaa0 2024-03-29T04:09:33.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 -- 172.21.15.156:0/672188257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 msgr2=0x7ffaf815d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 --2- 172.21.15.156:0/672188257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815d350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ffaf806ce70 tx=0x7ffae002b910 comp rx=0 tx=0).stop 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 -- 172.21.15.156:0/672188257 shutdown_connections 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 --2- 172.21.15.156:0/672188257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815d350 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 --2- 172.21.15.156:0/672188257 >> v2:172.21.15.203:3300/0 conn(0x7ffaf8156ac0 0x7ffaf80ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 -- 172.21.15.156:0/672188257 >> 172.21.15.156:0/672188257 conn(0x7ffaf8072ea0 msgr2=0x7ffaf806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 -- 172.21.15.156:0/672188257 shutdown_connections 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.292+0000 7ffafd790640 1 -- 172.21.15.156:0/672188257 wait complete. 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.293+0000 7ffafd790640 1 Processor -- start 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.293+0000 7ffafd790640 1 -- start start 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffafd790640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffafd790640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf80b5df0 0x7ffaf80b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffafd790640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffaf8156ac0 0x7ffaf814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffafd790640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf815cb50 con 0x7ffaf806eaa0 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffafd790640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf80cef10 con 0x7ffaf80b5df0 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffafd790640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ffaf815d890 con 0x7ffaf8156ac0 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43640/0 (socket says 172.21.15.156:43640) 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 -- 172.21.15.156:0/1992281338 learned_addr learned my addr 172.21.15.156:0/1992281338 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaee7fc640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf80b5df0 0x7ffaf80b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf77fe640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffaf8156ac0 0x7ffaf814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 -- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffaf8156ac0 msgr2=0x7ffaf814e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffaf8156ac0 0x7ffaf814e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 -- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf80b5df0 msgr2=0x7ffaf80b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf80b5df0 0x7ffaf80b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 -- 172.21.15.156:0/1992281338 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf80b4660 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaee7fc640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf80b5df0 0x7ffaf80b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf77fe640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffaf8156ac0 0x7ffaf814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.294+0000 7ffaf6ffd640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815b6f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffae800d830 tx=0x7ffae800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.295+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae800b6c0 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.295+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae8019070 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.295+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf80b4940 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.295+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae800ba80 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.295+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf814e1e0 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.296+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaf8073770 con 0x7ffaf806eaa0 2024-03-29T04:09:33.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.296+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ffae800bc20 con 0x7ffaf806eaa0 2024-03-29T04:09:33.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.297+0000 7ffaeffff640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ffad806c560 0x7ffad806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.297+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffae808bb70 con 0x7ffaf806eaa0 2024-03-29T04:09:33.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.297+0000 7ffaee7fc640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ffad806c560 0x7ffad806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.279 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.297+0000 7ffaee7fc640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ffad806c560 0x7ffad806ea20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffae0036000 tx=0x7ffae003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.301+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ffae801c360 con 0x7ffaf806eaa0 2024-03-29T04:09:33.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.482+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffaf804f9d0 con 0x7ffaf806eaa0 2024-03-29T04:09:33.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.482+0000 7ffaeffff640 1 -- 172.21.15.156:0/1992281338 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ffae805a190 con 0x7ffaf806eaa0 2024-03-29T04:09:33.462 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ffad806c560 msgr2=0x7ffad806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ffad806c560 0x7ffad806ea20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffae0036000 tx=0x7ffae003c040 comp rx=0 tx=0).stop 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 msgr2=0x7ffaf815b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815b6f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffae800d830 tx=0x7ffae800dcf0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 shutdown_connections 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ffad806c560 0x7ffad806ea20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffaf8156ac0 0x7ffaf814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf80b5df0 0x7ffaf80b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 --2- 172.21.15.156:0/1992281338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffaf806eaa0 0x7ffaf815b6f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 >> 172.21.15.156:0/1992281338 conn(0x7ffaf8072ea0 msgr2=0x7ffaf806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:33.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 shutdown_connections 2024-03-29T04:09:33.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.485+0000 7ffafd790640 1 -- 172.21.15.156:0/1992281338 wait complete. 2024-03-29T04:09:33.476 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:33.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.617+0000 7f94e8633640 1 Processor -- start 2024-03-29T04:09:33.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.617+0000 7f94e8633640 1 -- start start 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e8633640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 0x7f94e00b4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e8633640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e015c240 con 0x7f94e006f8d0 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e8633640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e015c520 con 0x7f94e015b730 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e8633640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f94e0159200 con 0x7f94e0158e20 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e6bd0640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f94e006f8d0 0x7f94e00cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33218/0 (socket says 172.21.15.156:33218) 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e6bd0640 1 -- 172.21.15.156:0/3176383652 learned_addr learned my addr 172.21.15.156:0/3176383652 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e63cf640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 0x7f94e00b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988917912 0 0) 0x7f94e015c240 con 0x7f94e006f8d0 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94cc003520 con 0x7f94e006f8d0 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1673163078 0 0) 0x7f94cc003520 con 0x7f94e006f8d0 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 msgr2=0x7f94e00b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 0x7f94e00b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 >> v1:172.21.15.203:6789/0 conn(0x7f94e015b730 legacy=0x7f94e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e00b3c60 con 0x7f94e006f8d0 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f94d80019f0 con 0x7f94e006f8d0 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 >> v1:172.21.15.156:6789/0 conn(0x7f94e006f8d0 legacy=0x7f94e00cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 0x7f94cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc004d40 0x7f94cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94e0159200 con 0x7f94cc0039e0 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc00ba70 con 0x7f94cc004d40 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.619+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f94cc00bc70 con 0x7f94e0158e20 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e63cf640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e6bd0640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc004d40 0x7f94cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e5bce640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 0x7f94cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e63cf640 1 -- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 msgr2=0x7f94cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e63cf640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e0158e20 0x7f94cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e63cf640 1 -- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc004d40 msgr2=0x7f94cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e63cf640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc004d40 0x7f94cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.620+0000 7f94e63cf640 1 -- 172.21.15.156:0/3176383652 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94cc003520 con 0x7f94cc0039e0 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e63cf640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94cc003e00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009a90 tx=0x7f94cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc0f5c2778c5124 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc039070 con 0x7f94cc0039e0 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94dc02c680 con 0x7f94cc0039e0 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e53cd640 1 -- 172.21.15.156:0/3176383652 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc009690 con 0x7f94cc0039e0 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e8633640 1 -- 172.21.15.156:0/3176383652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 msgr2=0x7f94cc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e8633640 1 --2- 172.21.15.156:0/3176383652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94cc003e00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009a90 tx=0x7f94cc005830 comp rx=0 tx=0).stop 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e6bd0640 1 -- 172.21.15.156:0/3176383652 reap_dead start 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e8633640 1 -- 172.21.15.156:0/3176383652 shutdown_connections 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.621+0000 7f94e8633640 1 -- 172.21.15.156:0/3176383652 >> 172.21.15.156:0/3176383652 conn(0x7f94e0072ec0 msgr2=0x7f94e006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.622+0000 7f94e8633640 1 -- 172.21.15.156:0/3176383652 shutdown_connections 2024-03-29T04:09:33.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.622+0000 7f94e8633640 1 -- 172.21.15.156:0/3176383652 wait complete. 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.623+0000 7f94e8633640 1 Processor -- start 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.623+0000 7f94e8633640 1 -- start start 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.623+0000 7f94e8633640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94e01e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e01f57d0 0x7f94e00b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e00b1f50 0x7f94e01573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94e015bbb0 con 0x7f94cc0039e0 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94e00b3fd0 con 0x7f94e01f57d0 2024-03-29T04:09:33.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f94e015bd30 con 0x7f94e00b1f50 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94e01e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94e01e58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43652/0 (socket says 172.21.15.156:43652) 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 -- 172.21.15.156:0/586876486 learned_addr learned my addr 172.21.15.156:0/586876486 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e5bce640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e01f57d0 0x7f94e00b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e6bd0640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e00b1f50 0x7f94e01573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 -- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e00b1f50 msgr2=0x7f94e01573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e00b1f50 0x7f94e01573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 -- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e01f57d0 msgr2=0x7f94e00b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e01f57d0 0x7f94e00b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 -- 172.21.15.156:0/586876486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e00ada50 con 0x7f94cc0039e0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e63cf640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94e01e58b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009a90 tx=0x7f94dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc039050 con 0x7f94cc0039e0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94dc045070 con 0x7f94cc0039e0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e00add30 con 0x7f94cc0039e0 2024-03-29T04:09:33.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc003850 con 0x7f94cc0039e0 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.624+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e00ae2f0 con 0x7f94cc0039e0 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.625+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e0073790 con 0x7f94cc0039e0 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.626+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f94dc0039f0 con 0x7f94cc0039e0 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.627+0000 7f94d6ffd640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94c406c530 0x7f94c406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.627+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f94dc0ad230 con 0x7f94cc0039e0 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.627+0000 7f94e5bce640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94c406c530 0x7f94c406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.627+0000 7f94e5bce640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94c406c530 0x7f94c406e9f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94e006f6a0 tx=0x7f94d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.631+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f94dc04e020 con 0x7f94cc0039e0 2024-03-29T04:09:33.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.820+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f94e004f9d0 con 0x7f94cc0039e0 2024-03-29T04:09:33.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.820+0000 7f94d6ffd640 1 -- 172.21.15.156:0/586876486 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f94dc07b910 con 0x7f94cc0039e0 2024-03-29T04:09:33.800 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94c406c530 msgr2=0x7f94c406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94c406c530 0x7f94c406e9f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94e006f6a0 tx=0x7f94d0009040 comp rx=0 tx=0).stop 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 msgr2=0x7f94e01e58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94e01e58b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009a90 tx=0x7f94dc004730 comp rx=0 tx=0).stop 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 shutdown_connections 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94c406c530 0x7f94c406e9f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94e00b1f50 0x7f94e01573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e01f57d0 0x7f94e00b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 --2- 172.21.15.156:0/586876486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94cc0039e0 0x7f94e01e58b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 >> 172.21.15.156:0/586876486 conn(0x7f94e0072ec0 msgr2=0x7f94e006e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 shutdown_connections 2024-03-29T04:09:33.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.823+0000 7f94e8633640 1 -- 172.21.15.156:0/586876486 wait complete. 2024-03-29T04:09:33.814 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:33.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.944+0000 7faa5222a640 1 Processor -- start 2024-03-29T04:09:33.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.944+0000 7faa5222a640 1 -- start start 2024-03-29T04:09:33.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa5222a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faa4c070480 0x7faa4c06f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa5222a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c0b5ed0 con 0x7faa4c06b5b0 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa5222a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c09ac90 con 0x7faa4c15e960 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa5222a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7faa4c09aa40 con 0x7faa4c070480 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4bfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7faa4c06b5b0 0x7faa4c0ae040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33232/0 (socket says 172.21.15.156:33232) 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4bfff640 1 -- 172.21.15.156:0/2150872912 learned_addr learned my addr 172.21.15.156:0/2150872912 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695251061 0 0) 0x7faa4c0b5ed0 con 0x7faa4c06b5b0 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa34003520 con 0x7faa4c06b5b0 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4affd640 1 --2- 172.21.15.156:0/2150872912 >> v2:172.21.15.203:3300/0 conn(0x7faa4c070480 0x7faa4c06f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1133285081 0 0) 0x7faa34003520 con 0x7faa4c06b5b0 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 >> v2:172.21.15.203:3300/0 conn(0x7faa4c070480 msgr2=0x7faa4c06f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 --2- 172.21.15.156:0/2150872912 >> v2:172.21.15.203:3300/0 conn(0x7faa4c070480 0x7faa4c06f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 >> v1:172.21.15.203:6789/0 conn(0x7faa4c15e960 legacy=0x7faa4c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0b6170 con 0x7faa4c06b5b0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.945+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7faa440019f0 con 0x7faa4c06b5b0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 >> v1:172.21.15.156:6789/0 conn(0x7faa4c06b5b0 legacy=0x7faa4c0ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa34004800 0x7faa34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa340051c0 0x7faa3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c09aa40 con 0x7faa340039b0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa3400c110 con 0x7faa340051c0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7faa3400c310 con 0x7faa34004800 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4bfff640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa340051c0 0x7faa3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4affd640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa34004800 0x7faa34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4b7fe640 1 -- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa34004800 msgr2=0x7faa34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa34004800 0x7faa34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4b7fe640 1 -- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa340051c0 msgr2=0x7faa3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa340051c0 0x7faa3400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4b7fe640 1 -- 172.21.15.156:0/2150872912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa34003520 con 0x7faa340039b0 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.946+0000 7faa4bfff640 1 -- 172.21.15.156:0/2150872912 reap_dead start 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa34003dd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faa3c001750 tx=0x7faa34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88585a1e9fa9ffb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c001370 con 0x7faa340039b0 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa3c0052d0 con 0x7faa340039b0 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa4a7fc640 1 -- 172.21.15.156:0/2150872912 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c003830 con 0x7faa340039b0 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa5222a640 1 -- 172.21.15.156:0/2150872912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 msgr2=0x7faa34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa5222a640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa34003dd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faa3c001750 tx=0x7faa34005830 comp rx=0 tx=0).stop 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.947+0000 7faa5222a640 1 -- 172.21.15.156:0/2150872912 shutdown_connections 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.948+0000 7faa5222a640 1 --2- 172.21.15.156:0/2150872912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa34003dd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.948+0000 7faa5222a640 1 -- 172.21.15.156:0/2150872912 >> 172.21.15.156:0/2150872912 conn(0x7faa4c072ce0 msgr2=0x7faa4c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.948+0000 7faa5222a640 1 -- 172.21.15.156:0/2150872912 shutdown_connections 2024-03-29T04:09:33.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.948+0000 7faa5222a640 1 -- 172.21.15.156:0/2150872912 wait complete. 2024-03-29T04:09:33.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.949+0000 7faa5222a640 1 Processor -- start 2024-03-29T04:09:33.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.949+0000 7faa5222a640 1 -- start start 2024-03-29T04:09:33.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa4c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa4c09bab0 0x7faa4c09bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa4c09e5d0 0x7faa4c09ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c15ed40 con 0x7faa340039b0 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0701f0 con 0x7faa4c09e5d0 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c06f900 con 0x7faa4c09bab0 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa4c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa4c155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43662/0 (socket says 172.21.15.156:43662) 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 -- 172.21.15.156:0/297850448 learned_addr learned my addr 172.21.15.156:0/297850448 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4bfff640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa4c09e5d0 0x7faa4c09ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4affd640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa4c09bab0 0x7faa4c09bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 -- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa4c09bab0 msgr2=0x7faa4c09bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa4c09bab0 0x7faa4c09bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 -- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa4c09e5d0 msgr2=0x7faa4c09ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:33.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa4c09e5d0 0x7faa4c09ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 -- 172.21.15.156:0/297850448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c1f4630 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4bfff640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa4c09e5d0 0x7faa4c09ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa4b7fe640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa4c155ed0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faa3c005040 tx=0x7faa3c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c041020 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa3c005480 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c1f4910 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.950+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c007a30 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.951+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c1f42d0 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.952+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c09f3c0 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.952+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7faa3c044020 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.952+0000 7faa33fff640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7faa3806c530 0x7faa3806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.952+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7faa3c0ad180 con 0x7faa340039b0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.952+0000 7faa4affd640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7faa3806c530 0x7faa3806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:33.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.953+0000 7faa4affd640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7faa3806c530 0x7faa3806e9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faa4c09d890 tx=0x7faa40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:33.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:33.957+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7faa3c04b030 con 0x7faa340039b0 2024-03-29T04:09:34.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.136+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa4c0b3480 con 0x7faa340039b0 2024-03-29T04:09:34.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.137+0000 7faa33fff640 1 -- 172.21.15.156:0/297850448 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7faa3c07b690 con 0x7faa340039b0 2024-03-29T04:09:34.116 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7faa3806c530 msgr2=0x7faa3806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7faa3806c530 0x7faa3806e9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faa4c09d890 tx=0x7faa40008040 comp rx=0 tx=0).stop 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 msgr2=0x7faa4c155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa4c155ed0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faa3c005040 tx=0x7faa3c001120 comp rx=0 tx=0).stop 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 shutdown_connections 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7faa3806c530 0x7faa3806e9f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa4c09e5d0 0x7faa4c09ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faa4c09bab0 0x7faa4c09bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 --2- 172.21.15.156:0/297850448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa340039b0 0x7faa4c155ed0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.139+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 >> 172.21.15.156:0/297850448 conn(0x7faa4c072ce0 msgr2=0x7faa4c15dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:34.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.140+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 shutdown_connections 2024-03-29T04:09:34.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.140+0000 7faa5222a640 1 -- 172.21.15.156:0/297850448 wait complete. 2024-03-29T04:09:34.130 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:34.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.255+0000 7f4275ce0640 1 Processor -- start 2024-03-29T04:09:34.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.255+0000 7f4275ce0640 1 -- start start 2024-03-29T04:09:34.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f4275ce0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f42700d27b0 0x7f4270167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f4275ce0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427015d960 con 0x7f4270071020 2024-03-29T04:09:34.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f4275ce0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427013f7f0 con 0x7f4270141ca0 2024-03-29T04:09:34.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f4275ce0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f427015d3a0 con 0x7f42700d27b0 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426f7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4270071020 0x7f4270071400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33234/0 (socket says 172.21.15.156:33234) 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426f7fe640 1 -- 172.21.15.156:0/201866502 learned_addr learned my addr 172.21.15.156:0/201866502 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426ffff640 1 --2- 172.21.15.156:0/201866502 >> v2:172.21.15.203:3300/0 conn(0x7f42700d27b0 0x7f4270167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426e7fc640 1 -- 172.21.15.156:0/201866502 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855472499 0 0) 0x7f427015d960 con 0x7f4270071020 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426e7fc640 1 -- 172.21.15.156:0/201866502 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4258003520 con 0x7f4270071020 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426ffff640 1 -- 172.21.15.156:0/201866502 >> v1:172.21.15.203:6789/0 conn(0x7f4270141ca0 legacy=0x7f427015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426ffff640 1 -- 172.21.15.156:0/201866502 >> v1:172.21.15.156:6789/0 conn(0x7f4270071020 legacy=0x7f4270071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426e7fc640 1 -- 172.21.15.156:0/201866502 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1080522258 0 0) 0x7f4258003520 con 0x7f4270071020 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.256+0000 7f426ffff640 1 -- 172.21.15.156:0/201866502 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427015d680 con 0x7f42700d27b0 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.257+0000 7f426ffff640 1 --2- 172.21.15.156:0/201866502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f4270167110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4268009d90 tx=0x7f426802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=707ad4c05e61bcc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.257+0000 7f426e7fc640 1 -- 172.21.15.156:0/201866502 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268039070 con 0x7f42700d27b0 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f426e7fc640 1 -- 172.21.15.156:0/201866502 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426802c940 con 0x7f42700d27b0 2024-03-29T04:09:34.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f426e7fc640 1 -- 172.21.15.156:0/201866502 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260001670 con 0x7f42700d27b0 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 -- 172.21.15.156:0/201866502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 msgr2=0x7f4270167110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 --2- 172.21.15.156:0/201866502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f4270167110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4268009d90 tx=0x7f426802be10 comp rx=0 tx=0).stop 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 -- 172.21.15.156:0/201866502 shutdown_connections 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 --2- 172.21.15.156:0/201866502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f4270167110 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 -- 172.21.15.156:0/201866502 >> 172.21.15.156:0/201866502 conn(0x7f42700b3eb0 msgr2=0x7f42700b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 -- 172.21.15.156:0/201866502 shutdown_connections 2024-03-29T04:09:34.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.258+0000 7f4275ce0640 1 -- 172.21.15.156:0/201866502 wait complete. 2024-03-29T04:09:34.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.259+0000 7f4275ce0640 1 Processor -- start 2024-03-29T04:09:34.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.259+0000 7f4275ce0640 1 -- start start 2024-03-29T04:09:34.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f4275ce0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 0x7f42700b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f4275ce0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f427015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f4275ce0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4270141ca0 0x7f427015b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f4275ce0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f427015d080 con 0x7f4270071020 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f4275ce0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42700d1280 con 0x7f42700d27b0 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f4275ce0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4270167650 con 0x7f4270141ca0 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 0x7f42700b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 0x7f42700b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43664/0 (socket says 172.21.15.156:43664) 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f426f7fe640 1 -- 172.21.15.156:0/1086772142 learned_addr learned my addr 172.21.15.156:0/1086772142 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f426effd640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f427015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.260+0000 7f426ffff640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4270141ca0 0x7f427015b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426f7fe640 1 -- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4270141ca0 msgr2=0x7f427015b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426f7fe640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4270141ca0 0x7f427015b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426f7fe640 1 -- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 msgr2=0x7f427015b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426f7fe640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f427015b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426f7fe640 1 -- 172.21.15.156:0/1086772142 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4270145c80 con 0x7f4270071020 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426effd640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f427015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:34.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f426f7fe640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 0x7f42700b66e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f426000c760 tx=0x7f426000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260017070 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4260005af0 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427015bd20 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260007070 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.261+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427015c2e0 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.262+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4270070e10 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.262+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f426001a070 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.263+0000 7f4257fff640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f425c06c530 0x7f425c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.263+0000 7f426effd640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f425c06c530 0x7f425c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.263+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f426008ace0 con 0x7f4270071020 2024-03-29T04:09:34.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.263+0000 7f426effd640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f425c06c530 0x7f425c06e9f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f427015ce60 tx=0x7f4264009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.268+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4260059280 con 0x7f4270071020 2024-03-29T04:09:34.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.461+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f427015c520 con 0x7f4270071020 2024-03-29T04:09:34.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.461+0000 7f4257fff640 1 -- 172.21.15.156:0/1086772142 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f4260058b80 con 0x7f4270071020 2024-03-29T04:09:34.440 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f425c06c530 msgr2=0x7f425c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f425c06c530 0x7f425c06e9f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f427015ce60 tx=0x7f4264009040 comp rx=0 tx=0).stop 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 msgr2=0x7f42700b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 0x7f42700b66e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f426000c760 tx=0x7f426000cc20 comp rx=0 tx=0).stop 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 shutdown_connections 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f425c06c530 0x7f425c06e9f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4270141ca0 0x7f427015b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42700d27b0 0x7f427015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 --2- 172.21.15.156:0/1086772142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4270071020 0x7f42700b66e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 >> 172.21.15.156:0/1086772142 conn(0x7f42700b3eb0 msgr2=0x7f427015f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:34.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 shutdown_connections 2024-03-29T04:09:34.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.463+0000 7f4275ce0640 1 -- 172.21.15.156:0/1086772142 wait complete. 2024-03-29T04:09:34.453 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:09:34.454 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:34.469 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:34.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.631+0000 7f9620959640 1 Processor -- start 2024-03-29T04:09:34.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.631+0000 7f9620959640 1 -- start start 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f9620959640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f961c06dbc0 0x7f961c169350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f9620959640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f961c0cfa20 con 0x7f961c15e730 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f9620959640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f961c0cfd00 con 0x7f961c0d42c0 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f9620959640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f961c06c720 con 0x7f961c06dbc0 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f961a59c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f961c15e730 0x7f961c0ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33248/0 (socket says 172.21.15.156:33248) 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f961a59c640 1 -- 172.21.15.156:0/4186455387 learned_addr learned my addr 172.21.15.156:0/4186455387 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.632+0000 7f961ad9d640 1 --2- 172.21.15.156:0/4186455387 >> v2:172.21.15.203:3300/0 conn(0x7f961c06dbc0 0x7f961c169350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024199795 0 0) 0x7f961c0cfa20 con 0x7f961c15e730 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9608003520 con 0x7f961c15e730 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522265750 0 0) 0x7f961c0cfd00 con 0x7f961c0d42c0 2024-03-29T04:09:34.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f961c0cfa20 con 0x7f961c0d42c0 2024-03-29T04:09:34.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f961ad9d640 1 -- 172.21.15.156:0/4186455387 >> v1:172.21.15.203:6789/0 conn(0x7f961c0d42c0 legacy=0x7f961c0d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3227426722 0 0) 0x7f9608003520 con 0x7f961c15e730 2024-03-29T04:09:34.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f961ad9d640 1 -- 172.21.15.156:0/4186455387 >> v1:172.21.15.156:6789/0 conn(0x7f961c15e730 legacy=0x7f961c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.633+0000 7f961ad9d640 1 -- 172.21.15.156:0/4186455387 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c0cffe0 con 0x7f961c06dbc0 2024-03-29T04:09:34.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.634+0000 7f961ad9d640 1 --2- 172.21.15.156:0/4186455387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c06dbc0 0x7f961c169350 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9614009d90 tx=0x7f961402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a86e7ed33affd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.634+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614039070 con 0x7f961c06dbc0 2024-03-29T04:09:34.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.634+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961402c940 con 0x7f961c06dbc0 2024-03-29T04:09:34.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.634+0000 7f9619d9b640 1 -- 172.21.15.156:0/4186455387 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c001670 con 0x7f961c06dbc0 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 -- 172.21.15.156:0/4186455387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c06dbc0 msgr2=0x7f961c169350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 --2- 172.21.15.156:0/4186455387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c06dbc0 0x7f961c169350 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9614009d90 tx=0x7f961402be10 comp rx=0 tx=0).stop 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 -- 172.21.15.156:0/4186455387 shutdown_connections 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 --2- 172.21.15.156:0/4186455387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c06dbc0 0x7f961c169350 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 -- 172.21.15.156:0/4186455387 >> 172.21.15.156:0/4186455387 conn(0x7f961c08bfd0 msgr2=0x7f961c09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 -- 172.21.15.156:0/4186455387 shutdown_connections 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.635+0000 7f9620959640 1 -- 172.21.15.156:0/4186455387 wait complete. 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.636+0000 7f9620959640 1 Processor -- start 2024-03-29T04:09:34.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.636+0000 7f9620959640 1 -- start start 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9620959640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 0x7f961c161a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9620959640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f961c0d42c0 0x7f961c161300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9620959640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c15e730 0x7f961c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9620959640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f961c15eb10 con 0x7f961c06dbc0 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9620959640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f961c0d2220 con 0x7f961c15e730 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9620959640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f961c15ec90 con 0x7f961c0d42c0 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 0x7f961c161a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 0x7f961c161a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43674/0 (socket says 172.21.15.156:43674) 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 -- 172.21.15.156:0/3184679732 learned_addr learned my addr 172.21.15.156:0/3184679732 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961ad9d640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c15e730 0x7f961c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9613fff640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f961c0d42c0 0x7f961c161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 -- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f961c0d42c0 msgr2=0x7f961c161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f961c0d42c0 0x7f961c161300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 -- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c15e730 msgr2=0x7f961c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c15e730 0x7f961c156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 -- 172.21.15.156:0/3184679732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c161840 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961ad9d640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c15e730 0x7f961c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f9613fff640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f961c0d42c0 0x7f961c161300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.637+0000 7f961a59c640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 0x7f961c161a70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f960c00c780 tx=0x7f960c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.638+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c005ad0 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.638+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f960c005c70 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.638+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f961c0cbe90 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.638+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c0072b0 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.638+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f961c0cc450 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.639+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f961c068610 con 0x7f961c06dbc0 2024-03-29T04:09:34.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.639+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f960c007450 con 0x7f961c06dbc0 2024-03-29T04:09:34.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.639+0000 7f9612ffd640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f95fc06c530 0x7f95fc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.640+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f960c08b5c0 con 0x7f961c06dbc0 2024-03-29T04:09:34.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.640+0000 7f9613fff640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f95fc06c530 0x7f95fc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.640+0000 7f9613fff640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f95fc06c530 0x7f95fc06e9f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f961c156cb0 tx=0x7f9604009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.644+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f960c019300 con 0x7f961c06dbc0 2024-03-29T04:09:34.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.827+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f961c0b1e40 con 0x7f961c06dbc0 2024-03-29T04:09:34.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.828+0000 7f9612ffd640 1 -- 172.21.15.156:0/3184679732 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16878 (secure 0 0 0) 0x7f960c059ba0 con 0x7f961c06dbc0 2024-03-29T04:09:34.808 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:34.808 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":22,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:33.925733+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:34.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f95fc06c530 msgr2=0x7f95fc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f95fc06c530 0x7f95fc06e9f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f961c156cb0 tx=0x7f9604009040 comp rx=0 tx=0).stop 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 msgr2=0x7f961c161a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 0x7f961c161a70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f960c00c780 tx=0x7f960c00cc40 comp rx=0 tx=0).stop 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 shutdown_connections 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f95fc06c530 0x7f95fc06e9f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f961c15e730 0x7f961c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f961c0d42c0 0x7f961c161300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 --2- 172.21.15.156:0/3184679732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f961c06dbc0 0x7f961c161a70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 >> 172.21.15.156:0/3184679732 conn(0x7f961c08bfd0 msgr2=0x7f961c0b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 shutdown_connections 2024-03-29T04:09:34.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.831+0000 7f9620959640 1 -- 172.21.15.156:0/3184679732 wait complete. 2024-03-29T04:09:34.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:34.822 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:34.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.951+0000 7f0256d15640 1 Processor -- start 2024-03-29T04:09:34.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.951+0000 7f0256d15640 1 -- start start 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0256d15640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f02500d2a00 0x7f02500cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0256d15640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f02500cfd60 0x7f025006ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0256d15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02500d2fd0 con 0x7f025015e540 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0256d15640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f025006c970 con 0x7f02500cfd60 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0256d15640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f02500d32b0 con 0x7f02500d2a00 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f02552b2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f02500cfd60 0x7f025006ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0254ab1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f025015e540 0x7f02500cf110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33262/0 (socket says 172.21.15.156:33262) 2024-03-29T04:09:34.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f0254ab1640 1 -- 172.21.15.156:0/4194431147 learned_addr learned my addr 172.21.15.156:0/4194431147 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:34.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f02552b2640 1 -- 172.21.15.156:0/4194431147 >> v2:172.21.15.203:3300/0 conn(0x7f02500d2a00 msgr2=0x7f02500cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:34.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f02552b2640 1 --2- 172.21.15.156:0/4194431147 >> v2:172.21.15.203:3300/0 conn(0x7f02500d2a00 0x7f02500cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f02552b2640 1 -- 172.21.15.156:0/4194431147 >> v1:172.21.15.156:6789/0 conn(0x7f025015e540 legacy=0x7f02500cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.952+0000 7f024f7fe640 1 -- 172.21.15.156:0/4194431147 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688318124 0 0) 0x7f02500d2fd0 con 0x7f025015e540 2024-03-29T04:09:34.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.953+0000 7f02552b2640 1 -- 172.21.15.156:0/4194431147 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025006c600 con 0x7f02500cfd60 2024-03-29T04:09:34.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.953+0000 7f02552b2640 1 --2- 172.21.15.156:0/4194431147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500cfd60 0x7f025006ce00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f02500b7040 tx=0x7f024802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a27811da6ab1ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.953+0000 7f024f7fe640 1 -- 172.21.15.156:0/4194431147 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248039070 con 0x7f02500cfd60 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.953+0000 7f024f7fe640 1 -- 172.21.15.156:0/4194431147 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f024802cc80 con 0x7f02500cfd60 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.953+0000 7f024f7fe640 1 -- 172.21.15.156:0/4194431147 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248034660 con 0x7f02500cfd60 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 -- 172.21.15.156:0/4194431147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500cfd60 msgr2=0x7f025006ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 --2- 172.21.15.156:0/4194431147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500cfd60 0x7f025006ce00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f02500b7040 tx=0x7f024802bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 -- 172.21.15.156:0/4194431147 shutdown_connections 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 --2- 172.21.15.156:0/4194431147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500cfd60 0x7f025006ce00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 --2- 172.21.15.156:0/4194431147 >> v2:172.21.15.203:3300/0 conn(0x7f02500d2a00 0x7f02500cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 -- 172.21.15.156:0/4194431147 >> 172.21.15.156:0/4194431147 conn(0x7f02500d4450 msgr2=0x7f02500d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 -- 172.21.15.156:0/4194431147 shutdown_connections 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.954+0000 7f0256d15640 1 -- 172.21.15.156:0/4194431147 wait complete. 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.955+0000 7f0256d15640 1 Processor -- start 2024-03-29T04:09:34.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.955+0000 7f0256d15640 1 -- start start 2024-03-29T04:09:34.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0256d15640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02500cfd60 0x7f025014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0256d15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 0x7f02500b8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0256d15640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f025015e540 0x7f02500b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0256d15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f025015eaa0 con 0x7f02500d2a00 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0256d15640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f025015e920 con 0x7f02500cfd60 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0256d15640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f025006d340 con 0x7f025015e540 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 0x7f02500b8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 0x7f02500b8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43692/0 (socket says 172.21.15.156:43692) 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 -- 172.21.15.156:0/4218904546 learned_addr learned my addr 172.21.15.156:0/4218904546 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f0254ab1640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02500cfd60 0x7f025014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f02552b2640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f025015e540 0x7f02500b85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 -- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f025015e540 msgr2=0x7f02500b85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f025015e540 0x7f02500b85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 -- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02500cfd60 msgr2=0x7f025014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:34.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02500cfd60 0x7f025014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.956+0000 7f024ffff640 1 -- 172.21.15.156:0/4218904546 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02500cabb0 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.957+0000 7f024ffff640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 0x7f02500b8090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f023c00b4a0 tx=0x7f023c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.957+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c014070 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.957+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f023c010610 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.957+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c010a00 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.957+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0250161740 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.957+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f025014bb90 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.958+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02500a23a0 con 0x7f02500d2a00 2024-03-29T04:09:34.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.958+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f023c017070 con 0x7f02500d2a00 2024-03-29T04:09:34.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.959+0000 7f024d7fa640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f023406c530 0x7f023406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:34.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.959+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f023c08b180 con 0x7f02500d2a00 2024-03-29T04:09:34.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.959+0000 7f0254ab1640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f023406c530 0x7f023406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:34.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.959+0000 7f0254ab1640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f023406c530 0x7f023406e9f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f0244007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:34.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:34.963+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f023c0597b0 con 0x7f02500d2a00 2024-03-29T04:09:35.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.144+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f025004f9d0 con 0x7f02500d2a00 2024-03-29T04:09:35.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.144+0000 7f024d7fa640 1 -- 172.21.15.156:0/4218904546 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f023c0590b0 con 0x7f02500d2a00 2024-03-29T04:09:35.124 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:35.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f023406c530 msgr2=0x7f023406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f023406c530 0x7f023406e9f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f0244007c00 comp rx=0 tx=0).stop 2024-03-29T04:09:35.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 msgr2=0x7f02500b8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 0x7f02500b8090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f023c00b4a0 tx=0x7f023c00b960 comp rx=0 tx=0).stop 2024-03-29T04:09:35.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 shutdown_connections 2024-03-29T04:09:35.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f023406c530 0x7f023406e9f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f025015e540 0x7f02500b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02500d2a00 0x7f02500b8090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 --2- 172.21.15.156:0/4218904546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02500cfd60 0x7f025014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 >> 172.21.15.156:0/4218904546 conn(0x7f02500d4450 msgr2=0x7f02500b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:35.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 shutdown_connections 2024-03-29T04:09:35.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.146+0000 7f0256d15640 1 -- 172.21.15.156:0/4218904546 wait complete. 2024-03-29T04:09:35.140 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:35.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.265+0000 7f93b0d82640 1 Processor -- start 2024-03-29T04:09:35.245 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93b0d82640 1 -- start start 2024-03-29T04:09:35.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93b0d82640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f93ac0b5bd0 0x7f93ac1691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93b0d82640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac144b70 con 0x7f93ac143ec0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93b0d82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac141f60 con 0x7f93ac1415e0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93b0d82640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f93ac141d10 con 0x7f93ac0b5bd0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93a3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f93ac143ec0 0x7f93ac0b54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33264/0 (socket says 172.21.15.156:33264) 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.266+0000 7f93a3fff640 1 -- 172.21.15.156:0/3835080769 learned_addr learned my addr 172.21.15.156:0/3835080769 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422872435 0 0) 0x7f93ac144b70 con 0x7f93ac143ec0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9398003520 con 0x7f93ac143ec0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93aad9d640 1 --2- 172.21.15.156:0/3835080769 >> v2:172.21.15.203:3300/0 conn(0x7f93ac0b5bd0 0x7f93ac1691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2849496235 0 0) 0x7f9398003520 con 0x7f93ac143ec0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 >> v2:172.21.15.203:3300/0 conn(0x7f93ac0b5bd0 msgr2=0x7f93ac1691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 --2- 172.21.15.156:0/3835080769 >> v2:172.21.15.203:3300/0 conn(0x7f93ac0b5bd0 0x7f93ac1691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 >> v1:172.21.15.203:6789/0 conn(0x7f93ac1415e0 legacy=0x7f93ac144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac142240 con 0x7f93ac143ec0 2024-03-29T04:09:35.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f9394001670 con 0x7f93ac143ec0 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 >> v1:172.21.15.156:6789/0 conn(0x7f93ac143ec0 legacy=0x7f93ac0b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f93980039b0 0x7f9398003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9398004800 0x7f9398004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980052e0 0x7f939800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac141d10 con 0x7f9398004800 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f939800c1b0 con 0x7f93980052e0 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f939800c3b0 con 0x7f93980039b0 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a3fff640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9398004800 0x7f9398004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93aa59c640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f93980039b0 0x7f9398003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93aad9d640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980052e0 0x7f939800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a3fff640 1 -- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f93980039b0 msgr2=0x7f9398003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a3fff640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f93980039b0 0x7f9398003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a3fff640 1 -- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980052e0 msgr2=0x7f939800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a3fff640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980052e0 0x7f939800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93a3fff640 1 -- 172.21.15.156:0/3835080769 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9398003520 con 0x7f9398004800 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93aad9d640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980052e0 0x7f939800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.267+0000 7f93aad9d640 1 -- 172.21.15.156:0/3835080769 reap_dead start 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.268+0000 7f93a3fff640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9398004800 0x7f9398004c80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f939400c210 tx=0x7f9398005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c68917838f2a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.268+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394007cf0 con 0x7f9398004800 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.268+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9394030070 con 0x7f9398004800 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.268+0000 7f93a9d9b640 1 -- 172.21.15.156:0/3835080769 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394005d30 con 0x7f9398004800 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 -- 172.21.15.156:0/3835080769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9398004800 msgr2=0x7f9398004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9398004800 0x7f9398004c80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f939400c210 tx=0x7f9398005830 comp rx=0 tx=0).stop 2024-03-29T04:09:35.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 -- 172.21.15.156:0/3835080769 shutdown_connections 2024-03-29T04:09:35.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 --2- 172.21.15.156:0/3835080769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9398004800 0x7f9398004c80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 -- 172.21.15.156:0/3835080769 >> 172.21.15.156:0/3835080769 conn(0x7f93ac0b3eb0 msgr2=0x7f93ac0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:35.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 -- 172.21.15.156:0/3835080769 shutdown_connections 2024-03-29T04:09:35.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.269+0000 7f93b0d82640 1 -- 172.21.15.156:0/3835080769 wait complete. 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.270+0000 7f93b0d82640 1 Processor -- start 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.270+0000 7f93b0d82640 1 -- start start 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93b0d82640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9398004800 0x7f93ac140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93b0d82640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 0x7f93ac0d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93b0d82640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac0d16f0 0x7f93ac15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93b0d82640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac1419c0 con 0x7f93ac147030 2024-03-29T04:09:35.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93b0d82640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac142800 con 0x7f93ac0d16f0 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93b0d82640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f93ac06c6c0 con 0x7f9398004800 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93a3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 0x7f93ac0d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93a3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 0x7f93ac0d11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43708/0 (socket says 172.21.15.156:43708) 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.271+0000 7f93a3fff640 1 -- 172.21.15.156:0/31215055 learned_addr learned my addr 172.21.15.156:0/31215055 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93aad9d640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac0d16f0 0x7f93ac15ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93aa59c640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9398004800 0x7f93ac140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a3fff640 1 -- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9398004800 msgr2=0x7f93ac140ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a3fff640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9398004800 0x7f93ac140ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a3fff640 1 -- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac0d16f0 msgr2=0x7f93ac15ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a3fff640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac0d16f0 0x7f93ac15ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a3fff640 1 -- 172.21.15.156:0/31215055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac0d1c30 con 0x7f93ac147030 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a3fff640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 0x7f93ac0d11b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9394001970 tx=0x7f93ac142130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394045070 con 0x7f93ac147030 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9394001080 con 0x7f93ac147030 2024-03-29T04:09:35.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ac0d1f10 con 0x7f93ac147030 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394004df0 con 0x7f93ac147030 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.272+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ac0d24d0 con 0x7f93ac147030 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.273+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ac070e10 con 0x7f93ac147030 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.273+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9394007070 con 0x7f93ac147030 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.274+0000 7f93a2ffd640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f938c06c530 0x7f938c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.274+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f93940ac4f0 con 0x7f93ac147030 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.274+0000 7f93aa59c640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f938c06c530 0x7f938c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.274+0000 7f93aa59c640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f938c06c530 0x7f938c06e9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f939c000c00 tx=0x7f939c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.278+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f939407a440 con 0x7f93ac147030 2024-03-29T04:09:35.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.459+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f93ac04f9d0 con 0x7f93ac147030 2024-03-29T04:09:35.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.459+0000 7f93a2ffd640 1 -- 172.21.15.156:0/31215055 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f939407e740 con 0x7f93ac147030 2024-03-29T04:09:35.439 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:35.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f938c06c530 msgr2=0x7f938c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f938c06c530 0x7f938c06e9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f939c000c00 tx=0x7f939c009040 comp rx=0 tx=0).stop 2024-03-29T04:09:35.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 msgr2=0x7f93ac0d11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 0x7f93ac0d11b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9394001970 tx=0x7f93ac142130 comp rx=0 tx=0).stop 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 shutdown_connections 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f938c06c530 0x7f938c06e9f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac0d16f0 0x7f93ac15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93ac147030 0x7f93ac0d11b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 --2- 172.21.15.156:0/31215055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9398004800 0x7f93ac140ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 >> 172.21.15.156:0/31215055 conn(0x7f93ac0b3eb0 msgr2=0x7f93ac06db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 shutdown_connections 2024-03-29T04:09:35.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.461+0000 7f93b0d82640 1 -- 172.21.15.156:0/31215055 wait complete. 2024-03-29T04:09:35.452 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:35.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 Processor -- start 2024-03-29T04:09:35.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 -- start start 2024-03-29T04:09:35.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f368c0d2120 0x7f368c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c06cb10 0x7f368c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f368c0d2df0 con 0x7f368c15e920 2024-03-29T04:09:35.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f368c0cf7a0 con 0x7f368c0d2120 2024-03-29T04:09:35.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.592+0000 7f3690e7b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f368c0cf190 con 0x7f368c06cb10 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3683fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f368c0d2120 0x7f368c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f368a59c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f368c15e920 0x7f368c0d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33280/0 (socket says 172.21.15.156:33280) 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f368a59c640 1 -- 172.21.15.156:0/2756909055 learned_addr learned my addr 172.21.15.156:0/2756909055 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f368ad9d640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c06cb10 0x7f368c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3689d9b640 1 -- 172.21.15.156:0/2756909055 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940287805 0 0) 0x7f368c0d2df0 con 0x7f368c15e920 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3689d9b640 1 -- 172.21.15.156:0/2756909055 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3678003520 con 0x7f368c15e920 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3683fff640 1 -- 172.21.15.156:0/2756909055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c06cb10 msgr2=0x7f368c0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3683fff640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c06cb10 0x7f368c0cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3683fff640 1 -- 172.21.15.156:0/2756909055 >> v1:172.21.15.156:6789/0 conn(0x7f368c15e920 legacy=0x7f368c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f3683fff640 1 -- 172.21.15.156:0/2756909055 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c0d3080 con 0x7f368c0d2120 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.593+0000 7f368ad9d640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c06cb10 0x7f368c0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3683fff640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c0d2120 0x7f368c06c5d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f36740097a0 tx=0x7f367402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a375e488941090f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3689d9b640 1 -- 172.21.15.156:0/2756909055 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674035020 con 0x7f368c0d2120 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3689d9b640 1 -- 172.21.15.156:0/2756909055 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3674031830 con 0x7f368c0d2120 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3689d9b640 1 -- 172.21.15.156:0/2756909055 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674031c00 con 0x7f368c0d2120 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 -- 172.21.15.156:0/2756909055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c0d2120 msgr2=0x7f368c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c0d2120 0x7f368c06c5d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f36740097a0 tx=0x7f367402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 -- 172.21.15.156:0/2756909055 shutdown_connections 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c06cb10 0x7f368c0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2756909055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c0d2120 0x7f368c06c5d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 -- 172.21.15.156:0/2756909055 >> 172.21.15.156:0/2756909055 conn(0x7f368c09d7a0 msgr2=0x7f368c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 -- 172.21.15.156:0/2756909055 shutdown_connections 2024-03-29T04:09:35.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.594+0000 7f3690e7b640 1 -- 172.21.15.156:0/2756909055 wait complete. 2024-03-29T04:09:35.578 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 Processor -- start 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 -- start start 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 0x7f368c152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368c0d2120 0x7f368c0b7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c15e920 0x7f368c0b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f368c0d34e0 con 0x7f368c06cb10 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f368c0d3360 con 0x7f368c0d2120 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.596+0000 7f3690e7b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f368c0d0170 con 0x7f368c15e920 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 0x7f368c152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 0x7f368c152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43720/0 (socket says 172.21.15.156:43720) 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 -- 172.21.15.156:0/2509344273 learned_addr learned my addr 172.21.15.156:0/2509344273 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 -- 172.21.15.156:0/2509344273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c15e920 msgr2=0x7f368c0b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c15e920 0x7f368c0b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 -- 172.21.15.156:0/2509344273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368c0d2120 msgr2=0x7f368c0b7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368c0d2120 0x7f368c0b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 -- 172.21.15.156:0/2509344273 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c161b30 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f368a59c640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 0x7f368c152af0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c390 tx=0x7f367c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c017070 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f367c005ce0 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368c0b9830 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c0072d0 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.597+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f368c1617c0 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.598+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f368c0a2480 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.599+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f367c01a070 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.599+0000 7f3682ffd640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f366c06c530 0x7f366c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.599+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f367c08a9e0 con 0x7f368c06cb10 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.599+0000 7f3683fff640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f366c06c530 0x7f366c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.599+0000 7f3683fff640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f366c06c530 0x7f366c06e9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3674036000 tx=0x7f3674030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.603+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f367c058930 con 0x7f368c06cb10 2024-03-29T04:09:35.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.874+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f368c0d1d80 con 0x7f368c06cb10 2024-03-29T04:09:35.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.875+0000 7f3682ffd640 1 -- 172.21.15.156:0/2509344273 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f367c05cc30 con 0x7f368c06cb10 2024-03-29T04:09:35.854 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:35.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f366c06c530 msgr2=0x7f366c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f366c06c530 0x7f366c06e9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3674036000 tx=0x7f3674030000 comp rx=0 tx=0).stop 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 msgr2=0x7f368c152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 0x7f368c152af0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c390 tx=0x7f367c00c850 comp rx=0 tx=0).stop 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 shutdown_connections 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f366c06c530 0x7f366c06e9f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f368c15e920 0x7f368c0b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368c0d2120 0x7f368c0b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 --2- 172.21.15.156:0/2509344273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f368c06cb10 0x7f368c152af0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 >> 172.21.15.156:0/2509344273 conn(0x7f368c09d7a0 msgr2=0x7f368c0b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 shutdown_connections 2024-03-29T04:09:35.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.877+0000 7f3690e7b640 1 -- 172.21.15.156:0/2509344273 wait complete. 2024-03-29T04:09:35.867 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:35.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 Processor -- start 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 -- start start 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4de806ee00 0x7f4de806f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4de806e640 con 0x7f4de815b900 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4de806e970 con 0x7f4de806ee00 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.994+0000 7f4dedf9d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4de80b4980 con 0x7f4de8155820 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4de7fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4de815b900 0x7f4de815c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33284/0 (socket says 172.21.15.156:33284) 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4de7fff640 1 -- 172.21.15.156:0/356407903 learned_addr learned my addr 172.21.15.156:0/356407903 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4ded79c640 1 --2- 172.21.15.156:0/356407903 >> v2:172.21.15.203:3300/0 conn(0x7f4de806ee00 0x7f4de806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4decf9b640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4de77fe640 1 -- 172.21.15.156:0/356407903 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740531037 0 0) 0x7f4de806e640 con 0x7f4de815b900 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4de77fe640 1 -- 172.21.15.156:0/356407903 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd8003520 con 0x7f4de815b900 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4ded79c640 1 -- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 msgr2=0x7f4de80b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:35.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4ded79c640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4ded79c640 1 -- 172.21.15.156:0/356407903 >> v1:172.21.15.156:6789/0 conn(0x7f4de815b900 legacy=0x7f4de815c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4de77fe640 1 -- 172.21.15.156:0/356407903 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 619769603 0 0) 0x7f4dd8003520 con 0x7f4de815b900 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4ded79c640 1 -- 172.21.15.156:0/356407903 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de80b4c60 con 0x7f4de806ee00 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.995+0000 7f4decf9b640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.996+0000 7f4ded79c640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de806f260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4de806eaf0 tx=0x7f4de002bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=967f29f69340be8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.996+0000 7f4de77fe640 1 -- 172.21.15.156:0/356407903 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0039070 con 0x7f4de806ee00 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.996+0000 7f4de77fe640 1 -- 172.21.15.156:0/356407903 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de002c8a0 con 0x7f4de806ee00 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.996+0000 7f4de77fe640 1 -- 172.21.15.156:0/356407903 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4001690 con 0x7f4de806ee00 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 -- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 msgr2=0x7f4de806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de806f260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4de806eaf0 tx=0x7f4de002bd70 comp rx=0 tx=0).stop 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 -- 172.21.15.156:0/356407903 shutdown_connections 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de806f260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/356407903 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 -- 172.21.15.156:0/356407903 >> 172.21.15.156:0/356407903 conn(0x7f4de806c6c0 msgr2=0x7f4de806caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:35.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 -- 172.21.15.156:0/356407903 shutdown_connections 2024-03-29T04:09:35.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.997+0000 7f4dedf9d640 1 -- 172.21.15.156:0/356407903 wait complete. 2024-03-29T04:09:35.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.998+0000 7f4dedf9d640 1 Processor -- start 2024-03-29T04:09:35.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.998+0000 7f4dedf9d640 1 -- start start 2024-03-29T04:09:35.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4dedf9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de80b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4dedf9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4dedf9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 0x7f4de80a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4dedf9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8155c00 con 0x7f4de815b900 2024-03-29T04:09:35.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4dedf9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4de80b4690 con 0x7f4de806ee00 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4dedf9d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4de815d020 con 0x7f4de8155820 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 0x7f4de80a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 0x7f4de80a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43722/0 (socket says 172.21.15.156:43722) 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 -- 172.21.15.156:0/3915115939 learned_addr learned my addr 172.21.15.156:0/3915115939 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4de7fff640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4decf9b640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de80b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 -- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 msgr2=0x7f4de80cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 -- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 msgr2=0x7f4de80b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de80b2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 -- 172.21.15.156:0/3915115939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de80cef40 con 0x7f4de815b900 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4decf9b640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de80b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:35.999+0000 7f4ded79c640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 0x7f4de80a94a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009e80 tx=0x7f4de0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.000+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0039050 con 0x7f4de815b900 2024-03-29T04:09:35.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.000+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de0045070 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.000+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0009450 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.000+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de80cf220 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.000+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de80cf7e0 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.001+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de804f8f0 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.001+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4de0040690 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.002+0000 7f4de57fa640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4dcc06c530 0x7f4dcc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.002+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4de00adfd0 con 0x7f4de815b900 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.002+0000 7f4decf9b640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4dcc06c530 0x7f4dcc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:35.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.002+0000 7f4decf9b640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4dcc06c530 0x7f4dcc06e9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc009ac0 tx=0x7f4ddc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:35.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.006+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4de007c600 con 0x7f4de815b900 2024-03-29T04:09:36.247 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.268+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4de80b65f0 con 0x7f4de815b900 2024-03-29T04:09:36.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.268+0000 7f4de57fa640 1 -- 172.21.15.156:0/3915115939 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f4de007bf00 con 0x7f4de815b900 2024-03-29T04:09:36.248 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:36.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4dcc06c530 msgr2=0x7f4dcc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4dcc06c530 0x7f4dcc06e9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc009ac0 tx=0x7f4ddc008040 comp rx=0 tx=0).stop 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 msgr2=0x7f4de80a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 0x7f4de80a94a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009e80 tx=0x7f4de0004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 shutdown_connections 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4dcc06c530 0x7f4dcc06e9f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4de815b900 0x7f4de80a94a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4de8155820 0x7f4de80cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 --2- 172.21.15.156:0/3915115939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de806ee00 0x7f4de80b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 >> 172.21.15.156:0/3915115939 conn(0x7f4de806c6c0 msgr2=0x7f4de8154e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 shutdown_connections 2024-03-29T04:09:36.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.271+0000 7f4dedf9d640 1 -- 172.21.15.156:0/3915115939 wait complete. 2024-03-29T04:09:36.261 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-03-29T04:09:36.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:36.261 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-29T04:09:36.288 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:36.288 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:09:36.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:36.289 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-29T04:09:36.343 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:09:36.343 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:36.362 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:09:36.362 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:36.420 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:36.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 Processor -- start 2024-03-29T04:09:36.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 -- start start 2024-03-29T04:09:36.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda5815e730 0x7fda580ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fda580cf3e0 0x7fda580d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda580d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fda5806c720 con 0x7fda5815e730 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fda580d4c70 con 0x7fda580cf3e0 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.543+0000 7fda5d21b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fda580cfa20 con 0x7fda580d4800 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda5815e730 0x7fda580ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda5815e730 0x7fda580ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43724/0 (socket says 172.21.15.156:43724) 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 -- 172.21.15.156:0/67529129 learned_addr learned my addr 172.21.15.156:0/67529129 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda4ffff640 1 --2- 172.21.15.156:0/67529129 >> v2:172.21.15.203:3300/0 conn(0x7fda580cf3e0 0x7fda580d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda5759e640 1 --2- 172.21.15.156:0/67529129 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda580d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 -- 172.21.15.156:0/67529129 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 msgr2=0x7fda580d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 --2- 172.21.15.156:0/67529129 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda580d0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 -- 172.21.15.156:0/67529129 >> v2:172.21.15.203:3300/0 conn(0x7fda580cf3e0 msgr2=0x7fda580d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:36.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 --2- 172.21.15.156:0/67529129 >> v2:172.21.15.203:3300/0 conn(0x7fda580cf3e0 0x7fda580d42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.544+0000 7fda56d9d640 1 -- 172.21.15.156:0/67529129 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda580cfd00 con 0x7fda5815e730 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda56d9d640 1 --2- 172.21.15.156:0/67529129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580ceea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fda580ba4e0 tx=0x7fda4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d889c7b71aa730 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5659c640 1 -- 172.21.15.156:0/67529129 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48035020 con 0x7fda5815e730 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5659c640 1 -- 172.21.15.156:0/67529129 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda48031870 con 0x7fda5815e730 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5659c640 1 -- 172.21.15.156:0/67529129 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48031c40 con 0x7fda5815e730 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 -- 172.21.15.156:0/67529129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 msgr2=0x7fda580ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 --2- 172.21.15.156:0/67529129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580ceea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fda580ba4e0 tx=0x7fda4802b910 comp rx=0 tx=0).stop 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 -- 172.21.15.156:0/67529129 shutdown_connections 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 --2- 172.21.15.156:0/67529129 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda580d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 --2- 172.21.15.156:0/67529129 >> v2:172.21.15.203:3300/0 conn(0x7fda580cf3e0 0x7fda580d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 --2- 172.21.15.156:0/67529129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580ceea0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 -- 172.21.15.156:0/67529129 >> 172.21.15.156:0/67529129 conn(0x7fda5808bfd0 msgr2=0x7fda5809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:36.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 -- 172.21.15.156:0/67529129 shutdown_connections 2024-03-29T04:09:36.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.545+0000 7fda5d21b640 1 -- 172.21.15.156:0/67529129 wait complete. 2024-03-29T04:09:36.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 Processor -- start 2024-03-29T04:09:36.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 -- start start 2024-03-29T04:09:36.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda580cf3e0 0x7fda580c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda5814bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda580d2220 con 0x7fda5815e730 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fda580d02c0 con 0x7fda580cf3e0 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5d21b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fda5815eb10 con 0x7fda580d4800 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43740/0 (socket says 172.21.15.156:43740) 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda5759e640 1 -- 172.21.15.156:0/1593353473 learned_addr learned my addr 172.21.15.156:0/1593353473 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.547+0000 7fda56d9d640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda580cf3e0 0x7fda580c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda4ffff640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda5814bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5759e640 1 -- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 msgr2=0x7fda5814bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5759e640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda5814bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5759e640 1 -- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda580cf3e0 msgr2=0x7fda580c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5759e640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda580cf3e0 0x7fda580c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5759e640 1 -- 172.21.15.156:0/1593353473 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda580c8770 con 0x7fda5815e730 2024-03-29T04:09:36.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5759e640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580c7d20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fda50007e70 tx=0x7fda5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50014620 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda50019070 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda580c8a50 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50014a20 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.548+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda580c9010 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.549+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda58068610 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.550+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fda50014bc0 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.550+0000 7fda4f7fe640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fda3806c530 0x7fda3806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.550+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fda5008be30 con 0x7fda5815e730 2024-03-29T04:09:36.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.550+0000 7fda56d9d640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fda3806c530 0x7fda3806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.551+0000 7fda56d9d640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fda3806c530 0x7fda3806e9f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fda48036000 tx=0x7fda4803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:36.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.555+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fda50059e10 con 0x7fda5815e730 2024-03-29T04:09:36.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.778+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda580b40a0 con 0x7fda5815e730 2024-03-29T04:09:36.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.780+0000 7fda4f7fe640 1 -- 172.21.15.156:0/1593353473 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16878 (secure 0 0 0) 0x7fda5005e110 con 0x7fda5815e730 2024-03-29T04:09:36.760 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:36.760 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":23,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:34.930341+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fda3806c530 msgr2=0x7fda3806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fda3806c530 0x7fda3806e9f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fda48036000 tx=0x7fda4803c040 comp rx=0 tx=0).stop 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 msgr2=0x7fda580c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580c7d20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fda50007e70 tx=0x7fda5000e4b0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 shutdown_connections 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fda3806c530 0x7fda3806e9f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda5815e730 0x7fda580c7d20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fda580d4800 0x7fda5814bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 --2- 172.21.15.156:0/1593353473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda580cf3e0 0x7fda580c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 >> 172.21.15.156:0/1593353473 conn(0x7fda5808bfd0 msgr2=0x7fda580b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:36.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 shutdown_connections 2024-03-29T04:09:36.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.782+0000 7fda5d21b640 1 -- 172.21.15.156:0/1593353473 wait complete. 2024-03-29T04:09:36.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:36.774 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:36.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.900+0000 7f6fe290e640 1 Processor -- start 2024-03-29T04:09:36.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.900+0000 7f6fe290e640 1 -- start start 2024-03-29T04:09:36.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe290e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6fdc15e920 0x7f6fdc0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe290e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6fdc0cfc30 0x7f6fdc0d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe290e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fdc0cf190 con 0x7f6fdc0d2120 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe290e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6fdc0cf7a0 con 0x7f6fdc0cfc30 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe290e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6fdc0d3080 con 0x7f6fdc15e920 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6fdc0cfc30 0x7f6fdc0d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6fdc0cfc30 0x7f6fdc0d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43750/0 (socket says 172.21.15.156:43750) 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 -- 172.21.15.156:0/4233830392 learned_addr learned my addr 172.21.15.156:0/4233830392 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fdbfff640 1 --2- 172.21.15.156:0/4233830392 >> v2:172.21.15.203:3300/0 conn(0x7f6fdc15e920 0x7f6fdc0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 -- 172.21.15.156:0/4233830392 >> v2:172.21.15.203:3300/0 conn(0x7f6fdc15e920 msgr2=0x7f6fdc0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 --2- 172.21.15.156:0/4233830392 >> v2:172.21.15.203:3300/0 conn(0x7f6fdc15e920 0x7f6fdc0d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 -- 172.21.15.156:0/4233830392 >> v1:172.21.15.203:6789/0 conn(0x7f6fdc0d2120 legacy=0x7f6fdc06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fe0eab640 1 -- 172.21.15.156:0/4233830392 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc0cf430 con 0x7f6fdc0cfc30 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.901+0000 7f6fdbfff640 1 --2- 172.21.15.156:0/4233830392 >> v2:172.21.15.203:3300/0 conn(0x7f6fdc15e920 0x7f6fdc0d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:36.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.902+0000 7f6fe0eab640 1 --2- 172.21.15.156:0/4233830392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0cfc30 0x7f6fdc0d00c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0b7040 tx=0x7f6fd402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfadd4695bfba4a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.902+0000 7f6fdaffd640 1 -- 172.21.15.156:0/4233830392 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd402ca80 con 0x7f6fdc0cfc30 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.902+0000 7f6fdaffd640 1 -- 172.21.15.156:0/4233830392 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd402cc20 con 0x7f6fdc0cfc30 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.902+0000 7f6fdaffd640 1 -- 172.21.15.156:0/4233830392 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40343f0 con 0x7f6fdc0cfc30 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 -- 172.21.15.156:0/4233830392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0cfc30 msgr2=0x7f6fdc0d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 --2- 172.21.15.156:0/4233830392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0cfc30 0x7f6fdc0d00c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0b7040 tx=0x7f6fd402bee0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 -- 172.21.15.156:0/4233830392 shutdown_connections 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 --2- 172.21.15.156:0/4233830392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0cfc30 0x7f6fdc0d00c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 --2- 172.21.15.156:0/4233830392 >> v2:172.21.15.203:3300/0 conn(0x7f6fdc15e920 0x7f6fdc0d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 -- 172.21.15.156:0/4233830392 >> 172.21.15.156:0/4233830392 conn(0x7f6fdc09d7a0 msgr2=0x7f6fdc0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 -- 172.21.15.156:0/4233830392 shutdown_connections 2024-03-29T04:09:36.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.903+0000 7f6fe290e640 1 -- 172.21.15.156:0/4233830392 wait complete. 2024-03-29T04:09:36.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.904+0000 7f6fe290e640 1 Processor -- start 2024-03-29T04:09:36.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.905+0000 7f6fe290e640 1 -- start start 2024-03-29T04:09:36.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe290e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6fdc0cfc30 0x7f6fdc14d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe290e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 0x7f6fdc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe290e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fdc15e920 0x7f6fdc0ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe290e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc0d34e0 con 0x7f6fdc0d2120 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe290e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc0d3360 con 0x7f6fdc15e920 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe290e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6fdc0d2c20 con 0x7f6fdc0cfc30 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 0x7f6fdc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 0x7f6fdc0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43756/0 (socket says 172.21.15.156:43756) 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 -- 172.21.15.156:0/380204635 learned_addr learned my addr 172.21.15.156:0/380204635 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 -- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6fdc0cfc30 msgr2=0x7f6fdc14d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdbfff640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6fdc0cfc30 0x7f6fdc14d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe0eab640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fdc15e920 0x7f6fdc0ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6fdc0cfc30 0x7f6fdc14d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 -- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fdc15e920 msgr2=0x7f6fdc0ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fdc15e920 0x7f6fdc0ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:36.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdb7fe640 1 -- 172.21.15.156:0/380204635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc0c8f70 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fe0eab640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fdc15e920 0x7f6fdc0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.906+0000 7f6fdbfff640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6fdc0cfc30 0x7f6fdc14d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.907+0000 7f6fdb7fe640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 0x7f6fdc0ce190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c460 tx=0x7f6fd000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.907+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0016020 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.907+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd0012870 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.907+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0012c60 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.907+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fdc0c9250 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.907+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fdc0c9810 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.908+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fdc0a2480 con 0x7f6fdc0d2120 2024-03-29T04:09:36.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.908+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6fd0019020 con 0x7f6fdc0d2120 2024-03-29T04:09:36.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.909+0000 7f6fd8ff9640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6fc806c530 0x7f6fc806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:36.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.909+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6fd008b420 con 0x7f6fdc0d2120 2024-03-29T04:09:36.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.909+0000 7f6fdbfff640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6fc806c530 0x7f6fc806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:36.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.909+0000 7f6fdbfff640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6fc806c530 0x7f6fc806e9f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc006fd0 tx=0x7f6fcc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:36.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:36.913+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6fd0020030 con 0x7f6fdc0d2120 2024-03-29T04:09:37.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.092+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6fdc04f9d0 con 0x7f6fdc0d2120 2024-03-29T04:09:37.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.093+0000 7f6fd8ff9640 1 -- 172.21.15.156:0/380204635 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f6fd0059a50 con 0x7f6fdc0d2120 2024-03-29T04:09:37.072 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:37.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6fc806c530 msgr2=0x7f6fc806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6fc806c530 0x7f6fc806e9f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc006fd0 tx=0x7f6fcc008040 comp rx=0 tx=0).stop 2024-03-29T04:09:37.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 msgr2=0x7f6fdc0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 0x7f6fdc0ce190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c460 tx=0x7f6fd000c920 comp rx=0 tx=0).stop 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 shutdown_connections 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6fc806c530 0x7f6fc806e9f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fdc15e920 0x7f6fdc0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0d2120 0x7f6fdc0ce190 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 --2- 172.21.15.156:0/380204635 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6fdc0cfc30 0x7f6fdc14d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.094+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 >> 172.21.15.156:0/380204635 conn(0x7f6fdc09d7a0 msgr2=0x7f6fdc06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.095+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 shutdown_connections 2024-03-29T04:09:37.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.095+0000 7f6fe290e640 1 -- 172.21.15.156:0/380204635 wait complete. 2024-03-29T04:09:37.085 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:37.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.214+0000 7fbc7a1db640 1 Processor -- start 2024-03-29T04:09:37.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.214+0000 7fbc7a1db640 1 -- start start 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc7a1db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc74071400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc7a1db640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbc74141ca0 0x7fbc7415c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc7a1db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc74071940 con 0x7fbc7415ceb0 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc7a1db640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbc7415d960 con 0x7fbc74141ca0 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc7a1db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7413f7f0 con 0x7fbc74071020 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc73fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbc7415ceb0 0x7fbc7415ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33296/0 (socket says 172.21.15.156:33296) 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc73fff640 1 -- 172.21.15.156:0/1786041885 learned_addr learned my addr 172.21.15.156:0/1786041885 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc737fe640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc74071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/1786041885 >> v2:172.21.15.203:3300/0 conn(0x7fbc74141ca0 0x7fbc7415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 959447135 0 0) 0x7fbc74071940 con 0x7fbc7415ceb0 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc60003520 con 0x7fbc7415ceb0 2024-03-29T04:09:37.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.215+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1876170981 0 0) 0x7fbc60003520 con 0x7fbc7415ceb0 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 msgr2=0x7fbc74071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc74071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 >> v2:172.21.15.203:3300/0 conn(0x7fbc74141ca0 msgr2=0x7fbc7415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 --2- 172.21.15.156:0/1786041885 >> v2:172.21.15.203:3300/0 conn(0x7fbc74141ca0 0x7fbc7415c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7415d3a0 con 0x7fbc7415ceb0 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fbc6c0019f0 con 0x7fbc7415ceb0 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/1786041885 >> v2:172.21.15.203:3300/0 conn(0x7fbc74141ca0 0x7fbc7415c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc737fe640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc74071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 >> v1:172.21.15.156:6789/0 conn(0x7fbc7415ceb0 legacy=0x7fbc7415ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc60003ef0 0x7fbc60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc60004d40 0x7fbc600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7415d960 con 0x7fbc60004d40 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7413f7f0 con 0x7fbc60003ef0 2024-03-29T04:09:37.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbc6000b9c0 con 0x7fbc74071020 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc73fff640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc60004d40 0x7fbc600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc60003ef0 0x7fbc60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc737fe640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc73fff640 1 -- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 msgr2=0x7fbc600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc73fff640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc74071020 0x7fbc600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc73fff640 1 -- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc60003ef0 msgr2=0x7fbc60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc73fff640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc60003ef0 0x7fbc60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.216+0000 7fbc73fff640 1 -- 172.21.15.156:0/1786041885 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc60003520 con 0x7fbc60004d40 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.217+0000 7fbc73fff640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc60004d40 0x7fbc600051c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00a8f0 tx=0x7fbc60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4c2f6ed44cd8f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.217+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00a4c0 con 0x7fbc60004d40 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.217+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c034650 con 0x7fbc60004d40 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.217+0000 7fbc72ffd640 1 -- 172.21.15.156:0/1786041885 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c034a80 con 0x7fbc60004d40 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc7a1db640 1 -- 172.21.15.156:0/1786041885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc60004d40 msgr2=0x7fbc600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/1786041885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc60004d40 0x7fbc600051c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00a8f0 tx=0x7fbc60005830 comp rx=0 tx=0).stop 2024-03-29T04:09:37.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc73fff640 1 -- 172.21.15.156:0/1786041885 reap_dead start 2024-03-29T04:09:37.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc7a1db640 1 -- 172.21.15.156:0/1786041885 shutdown_connections 2024-03-29T04:09:37.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc7a1db640 1 -- 172.21.15.156:0/1786041885 >> 172.21.15.156:0/1786041885 conn(0x7fbc740b3eb0 msgr2=0x7fbc740b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:37.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc7a1db640 1 -- 172.21.15.156:0/1786041885 shutdown_connections 2024-03-29T04:09:37.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.218+0000 7fbc7a1db640 1 -- 172.21.15.156:0/1786041885 wait complete. 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.219+0000 7fbc7a1db640 1 Processor -- start 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.219+0000 7fbc7a1db640 1 -- start start 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc7a1db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 0x7fbc74142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc7a1db640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc74156090 0x7fbc7415b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc7a1db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc7413a970 0x7fbc7415bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc7a1db640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc740d1280 con 0x7fbc74155cb0 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc7a1db640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc740d0fa0 con 0x7fbc74156090 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc7a1db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbc740d27b0 con 0x7fbc7413a970 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 0x7fbc74142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 0x7fbc74142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43778/0 (socket says 172.21.15.156:43778) 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 -- 172.21.15.156:0/410647992 learned_addr learned my addr 172.21.15.156:0/410647992 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:37.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 -- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc7413a970 msgr2=0x7fbc7415bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc73fff640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc7413a970 0x7fbc7415bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc74156090 0x7fbc7415b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc7413a970 0x7fbc7415bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 -- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc74156090 msgr2=0x7fbc7415b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc74156090 0x7fbc7415b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 -- 172.21.15.156:0/410647992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc74143400 con 0x7fbc74155cb0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc74156090 0x7fbc7415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.220+0000 7fbc737fe640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 0x7fbc74142cb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbc64011a00 tx=0x7fbc64011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.221+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6400faf0 con 0x7fbc74155cb0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.221+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6400fc90 con 0x7fbc74155cb0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.221+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc74143e90 con 0x7fbc74155cb0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.221+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64004030 con 0x7fbc74155cb0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.221+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7413bf70 con 0x7fbc74155cb0 2024-03-29T04:09:37.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.222+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc74070e10 con 0x7fbc74155cb0 2024-03-29T04:09:37.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.222+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbc64012040 con 0x7fbc74155cb0 2024-03-29T04:09:37.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.222+0000 7fbc70ff9640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbc5406c560 0x7fbc5406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.223+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbc5406c560 0x7fbc5406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.223+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbc6408bbe0 con 0x7fbc74155cb0 2024-03-29T04:09:37.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.223+0000 7fbc6bfff640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbc5406c560 0x7fbc5406ea20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbc74160b30 tx=0x7fbc5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.227+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbc64019070 con 0x7fbc74155cb0 2024-03-29T04:09:37.385 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.406+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbc7404f9d0 con 0x7fbc74155cb0 2024-03-29T04:09:37.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.407+0000 7fbc70ff9640 1 -- 172.21.15.156:0/410647992 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fbc6405a0f0 con 0x7fbc74155cb0 2024-03-29T04:09:37.386 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:37.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbc5406c560 msgr2=0x7fbc5406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbc5406c560 0x7fbc5406ea20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbc74160b30 tx=0x7fbc5c008040 comp rx=0 tx=0).stop 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 msgr2=0x7fbc74142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 0x7fbc74142cb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbc64011a00 tx=0x7fbc64011ec0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 shutdown_connections 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbc5406c560 0x7fbc5406ea20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbc7413a970 0x7fbc7415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc74156090 0x7fbc7415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 --2- 172.21.15.156:0/410647992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc74155cb0 0x7fbc74142cb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 >> 172.21.15.156:0/410647992 conn(0x7fbc740b3eb0 msgr2=0x7fbc7406f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 shutdown_connections 2024-03-29T04:09:37.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.409+0000 7fbc7a1db640 1 -- 172.21.15.156:0/410647992 wait complete. 2024-03-29T04:09:37.399 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:37.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.527+0000 7f99a38d3640 1 Processor -- start 2024-03-29T04:09:37.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.528+0000 7f99a38d3640 1 -- start start 2024-03-29T04:09:37.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.528+0000 7f99a38d3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f999c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.528+0000 7f99a38d3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f999c0d2df0 con 0x7f999c06cb10 2024-03-29T04:09:37.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.528+0000 7f99a38d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f999c0cf190 con 0x7f999c15e920 2024-03-29T04:09:37.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.528+0000 7f99a38d3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f999c0cf7a0 con 0x7f999c0d2120 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f99a1e70640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f999c06cb10 0x7f999c0cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33304/0 (socket says 172.21.15.156:33304) 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f99a1e70640 1 -- 172.21.15.156:0/2027380581 learned_addr learned my addr 172.21.15.156:0/2027380581 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f99a0e6e640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f999c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3420187352 0 0) 0x7f999c0d2df0 con 0x7f999c06cb10 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9988003520 con 0x7f999c06cb10 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 347005916 0 0) 0x7f9988003520 con 0x7f999c06cb10 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 msgr2=0x7f999c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f999c06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 >> v1:172.21.15.203:6789/0 conn(0x7f999c15e920 legacy=0x7f999c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c0d3080 con 0x7f999c06cb10 2024-03-29T04:09:37.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f99980019f0 con 0x7f999c06cb10 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f99a0e6e640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f999c06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 >> v1:172.21.15.156:6789/0 conn(0x7f999c06cb10 legacy=0x7f999c0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f99880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9988003f20 0x7f9988004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9988004d40 0x7f99880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f999c0cf7a0 con 0x7f9988004d40 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f998800ba20 con 0x7f9988003f20 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f998800bc20 con 0x7f999c0d2120 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f99a1e70640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9988004d40 0x7f99880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.529+0000 7f99a166f640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f99880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a0e6e640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9988003f20 0x7f9988004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a1e70640 1 -- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 msgr2=0x7f99880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a1e70640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c0d2120 0x7f99880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a1e70640 1 -- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9988003f20 msgr2=0x7f9988004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a1e70640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9988003f20 0x7f9988004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a1e70640 1 -- 172.21.15.156:0/2027380581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9988003520 con 0x7f9988004d40 2024-03-29T04:09:37.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.530+0000 7f99a1e70640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9988004d40 0x7f99880051c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f999800a8f0 tx=0x7f9988005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8cf0cbc9f9c4c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999800a4c0 con 0x7f9988004d40 2024-03-29T04:09:37.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9998034650 con 0x7f9988004d40 2024-03-29T04:09:37.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f9993fff640 1 -- 172.21.15.156:0/2027380581 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998034a80 con 0x7f9988004d40 2024-03-29T04:09:37.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a38d3640 1 -- 172.21.15.156:0/2027380581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9988004d40 msgr2=0x7f99880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a38d3640 1 --2- 172.21.15.156:0/2027380581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9988004d40 0x7f99880051c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f999800a8f0 tx=0x7f9988005830 comp rx=0 tx=0).stop 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a1e70640 1 -- 172.21.15.156:0/2027380581 reap_dead start 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a38d3640 1 -- 172.21.15.156:0/2027380581 shutdown_connections 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a38d3640 1 -- 172.21.15.156:0/2027380581 >> 172.21.15.156:0/2027380581 conn(0x7f999c09d7a0 msgr2=0x7f999c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a38d3640 1 -- 172.21.15.156:0/2027380581 shutdown_connections 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.531+0000 7f99a38d3640 1 -- 172.21.15.156:0/2027380581 wait complete. 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 Processor -- start 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 -- start start 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f999c0cd5a0 0x7f999c09e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 0x7f999c1e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c09f9b0 0x7f999c09fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f999c0d34e0 con 0x7f999c0cd980 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f999c0d3360 con 0x7f999c0cd5a0 2024-03-29T04:09:37.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.533+0000 7f99a38d3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f999c0d0340 con 0x7f999c09f9b0 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 0x7f999c1e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 0x7f999c1e8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43786/0 (socket says 172.21.15.156:43786) 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 -- 172.21.15.156:0/3928223063 learned_addr learned my addr 172.21.15.156:0/3928223063 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a166f640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f999c0cd5a0 0x7f999c09e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a1e70640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c09f9b0 0x7f999c09fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 -- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c09f9b0 msgr2=0x7f999c09fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c09f9b0 0x7f999c09fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 -- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f999c0cd5a0 msgr2=0x7f999c09e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f999c0cd5a0 0x7f999c09e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 -- 172.21.15.156:0/3928223063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c09e910 con 0x7f999c0cd980 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a1e70640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c09f9b0 0x7f999c09fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a0e6e640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 0x7f999c1e8d70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f998c0119d0 tx=0x7f998c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c00faf0 con 0x7f999c0cd980 2024-03-29T04:09:37.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f998c00fc90 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c004020 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.534+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999c09ebf0 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.535+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999c09f1b0 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.535+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f999c0a2480 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.536+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f998c012040 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.536+0000 7f9991ffb640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f998006c560 0x7f998006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.537+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f998c08bb60 con 0x7f999c0cd980 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.537+0000 7f99a166f640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f998006c560 0x7f998006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.537+0000 7f99a166f640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f998006c560 0x7f998006ea20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9994003700 tx=0x7f9994009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.541+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f998c019030 con 0x7f999c0cd980 2024-03-29T04:09:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.789+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f999c04f9d0 con 0x7f999c0cd980 2024-03-29T04:09:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.789+0000 7f9991ffb640 1 -- 172.21.15.156:0/3928223063 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f998c05a100 con 0x7f999c0cd980 2024-03-29T04:09:37.769 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f998006c560 msgr2=0x7f998006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f998006c560 0x7f998006ea20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9994003700 tx=0x7f9994009040 comp rx=0 tx=0).stop 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 msgr2=0x7f999c1e8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 0x7f999c1e8d70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f998c0119d0 tx=0x7f998c011e90 comp rx=0 tx=0).stop 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 shutdown_connections 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f998006c560 0x7f998006ea20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f999c09f9b0 0x7f999c09fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f999c0cd980 0x7f999c1e8d70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 --2- 172.21.15.156:0/3928223063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f999c0cd5a0 0x7f999c09e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 >> 172.21.15.156:0/3928223063 conn(0x7f999c09d7a0 msgr2=0x7f999c06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:37.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 shutdown_connections 2024-03-29T04:09:37.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.791+0000 7f99a38d3640 1 -- 172.21.15.156:0/3928223063 wait complete. 2024-03-29T04:09:37.782 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:37.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.905+0000 7f2e27a93640 1 Processor -- start 2024-03-29T04:09:37.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.905+0000 7f2e27a93640 1 -- start start 2024-03-29T04:09:37.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e27a93640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e2015e920 0x7f2e200d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e27a93640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e200d2120 0x7f2e2006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e27a93640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e200cf190 con 0x7f2e2006cb10 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e27a93640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2e200cf7a0 con 0x7f2e2015e920 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e27a93640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2e200d3080 con 0x7f2e200d2120 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e2015e920 0x7f2e200d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e2015e920 0x7f2e200d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43790/0 (socket says 172.21.15.156:43790) 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 -- 172.21.15.156:0/4253730921 learned_addr learned my addr 172.21.15.156:0/4253730921 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2502e640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e200d2120 0x7f2e2006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 -- 172.21.15.156:0/4253730921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e200d2120 msgr2=0x7f2e2006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e200d2120 0x7f2e2006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 -- 172.21.15.156:0/4253730921 >> v1:172.21.15.203:6789/0 conn(0x7f2e2006cb10 legacy=0x7f2e200cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2582f640 1 -- 172.21.15.156:0/4253730921 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e200cf430 con 0x7f2e2015e920 2024-03-29T04:09:37.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.906+0000 7f2e2502e640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e200d2120 0x7f2e2006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.907+0000 7f2e2582f640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2015e920 0x7f2e200d26e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009870 tx=0x7f2e1002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bacce0dad668127 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.907+0000 7f2e2482d640 1 -- 172.21.15.156:0/4253730921 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10035020 con 0x7f2e2015e920 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.907+0000 7f2e2482d640 1 -- 172.21.15.156:0/4253730921 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10031640 con 0x7f2e2015e920 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.907+0000 7f2e2482d640 1 -- 172.21.15.156:0/4253730921 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1002d040 con 0x7f2e2015e920 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 -- 172.21.15.156:0/4253730921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2015e920 msgr2=0x7f2e200d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2015e920 0x7f2e200d26e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009870 tx=0x7f2e1002b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 -- 172.21.15.156:0/4253730921 shutdown_connections 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e200d2120 0x7f2e2006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 --2- 172.21.15.156:0/4253730921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2015e920 0x7f2e200d26e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 -- 172.21.15.156:0/4253730921 >> 172.21.15.156:0/4253730921 conn(0x7f2e2009d7a0 msgr2=0x7f2e200d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 -- 172.21.15.156:0/4253730921 shutdown_connections 2024-03-29T04:09:37.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.908+0000 7f2e27a93640 1 -- 172.21.15.156:0/4253730921 wait complete. 2024-03-29T04:09:37.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.909+0000 7f2e27a93640 1 Processor -- start 2024-03-29T04:09:37.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.909+0000 7f2e27a93640 1 -- start start 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e27a93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 0x7f2e201545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e27a93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e200d2120 0x7f2e20160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e27a93640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e2015e920 0x7f2e2015f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e27a93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e200d04c0 con 0x7f2e2006cb10 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e27a93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e200d0340 con 0x7f2e200d2120 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e27a93640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2e200d2c20 con 0x7f2e2015e920 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 0x7f2e201545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 0x7f2e201545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43792/0 (socket says 172.21.15.156:43792) 2024-03-29T04:09:37.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 -- 172.21.15.156:0/3825789135 learned_addr learned my addr 172.21.15.156:0/3825789135 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 -- 172.21.15.156:0/3825789135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e2015e920 msgr2=0x7f2e2015f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e2015e920 0x7f2e2015f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 -- 172.21.15.156:0/3825789135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e200d2120 msgr2=0x7f2e20160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e200d2120 0x7f2e20160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 -- 172.21.15.156:0/3825789135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2015fcf0 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.910+0000 7f2e2582f640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 0x7f2e201545a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c00 tx=0x7f2e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.911+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10040070 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.911+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10039c10 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.911+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2015ffd0 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.911+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10035040 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.911+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e20160590 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.912+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e200d2a40 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.912+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2e10038020 con 0x7f2e2006cb10 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.912+0000 7f2e167fc640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e0406c530 0x7f2e0406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:37.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.913+0000 7f2e2502e640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e0406c530 0x7f2e0406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:37.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.913+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2e100ad1d0 con 0x7f2e2006cb10 2024-03-29T04:09:37.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.913+0000 7f2e2502e640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e0406c530 0x7f2e0406e9f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c006fd0 tx=0x7f2e1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:37.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:37.917+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2e200d2a40 con 0x7f2e2006cb10 2024-03-29T04:09:38.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.211+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2e2004f9d0 con 0x7f2e2006cb10 2024-03-29T04:09:38.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.211+0000 7f2e167fc640 1 -- 172.21.15.156:0/3825789135 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f2e1007b800 con 0x7f2e2006cb10 2024-03-29T04:09:38.190 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:38.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e0406c530 msgr2=0x7f2e0406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e0406c530 0x7f2e0406e9f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c006fd0 tx=0x7f2e1c008040 comp rx=0 tx=0).stop 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 msgr2=0x7f2e201545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 0x7f2e201545a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c00 tx=0x7f2e100046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 shutdown_connections 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2e0406c530 0x7f2e0406e9f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2e2015e920 0x7f2e2015f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e200d2120 0x7f2e20160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 --2- 172.21.15.156:0/3825789135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e2006cb10 0x7f2e201545a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 >> 172.21.15.156:0/3825789135 conn(0x7f2e2009d7a0 msgr2=0x7f2e200d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 shutdown_connections 2024-03-29T04:09:38.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.214+0000 7f2e27a93640 1 -- 172.21.15.156:0/3825789135 wait complete. 2024-03-29T04:09:38.204 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:09:38.205 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:09:38.224 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:38.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.347+0000 7fcbc4538640 1 Processor -- start 2024-03-29T04:09:38.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.347+0000 7fcbc4538640 1 -- start start 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc4538640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcbbc0cec00 0x7fcbbc1609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc4538640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcbbc0cc870 0x7fcbbc0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc4538640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbbc15cb50 con 0x7fcbbc1568f0 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc4538640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fcbbc0cf8b0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc4538640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcbbc0cf0b0 con 0x7fcbbc0cc870 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc22d4640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fcbbc1568f0 0x7fcbbc15d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33320/0 (socket says 172.21.15.156:33320) 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc22d4640 1 -- 172.21.15.156:0/1235583280 learned_addr learned my addr 172.21.15.156:0/1235583280 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc1ad3640 1 --2- 172.21.15.156:0/1235583280 >> v2:172.21.15.156:3300/0 conn(0x7fcbbc0cec00 0x7fcbbc1609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc2ad5640 1 --2- 172.21.15.156:0/1235583280 >> v2:172.21.15.203:3300/0 conn(0x7fcbbc0cc870 0x7fcbbc0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1235583280 >> v2:172.21.15.203:3300/0 conn(0x7fcbbc0cc870 msgr2=0x7fcbbc0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc1ad3640 1 --2- 172.21.15.156:0/1235583280 >> v2:172.21.15.203:3300/0 conn(0x7fcbbc0cc870 0x7fcbbc0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1235583280 >> v1:172.21.15.156:6789/0 conn(0x7fcbbc1568f0 legacy=0x7fcbbc15d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1235583280 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbbc0cf2f0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.348+0000 7fcbc12d2640 1 -- 172.21.15.156:0/1235583280 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556333921 0 0) 0x7fcbbc15cb50 con 0x7fcbbc1568f0 2024-03-29T04:09:38.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.349+0000 7fcbc1ad3640 1 --2- 172.21.15.156:0/1235583280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc1609b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0ce220 tx=0x7fcbac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705419c381813f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.349+0000 7fcbc12d2640 1 -- 172.21.15.156:0/1235583280 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0315a0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.349+0000 7fcbc12d2640 1 -- 172.21.15.156:0/1235583280 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbac037070 con 0x7fcbbc0cec00 2024-03-29T04:09:38.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.349+0000 7fcbc12d2640 1 -- 172.21.15.156:0/1235583280 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac031960 con 0x7fcbbc0cec00 2024-03-29T04:09:38.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 -- 172.21.15.156:0/1235583280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 msgr2=0x7fcbbc1609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1235583280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc1609b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0ce220 tx=0x7fcbac02b840 comp rx=0 tx=0).stop 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 -- 172.21.15.156:0/1235583280 shutdown_connections 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1235583280 >> v2:172.21.15.203:3300/0 conn(0x7fcbbc0cc870 0x7fcbbc0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1235583280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc1609b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 -- 172.21.15.156:0/1235583280 >> 172.21.15.156:0/1235583280 conn(0x7fcbbc0b3eb0 msgr2=0x7fcbbc0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 -- 172.21.15.156:0/1235583280 shutdown_connections 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.350+0000 7fcbc4538640 1 -- 172.21.15.156:0/1235583280 wait complete. 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.351+0000 7fcbc4538640 1 Processor -- start 2024-03-29T04:09:38.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.351+0000 7fcbc4538640 1 -- start start 2024-03-29T04:09:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.351+0000 7fcbc4538640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcbbc0cc870 0x7fcbbc1f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbbc1568f0 0x7fcbbc1ede20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc15cdf0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc0cd1e0 con 0x7fcbbc1568f0 2024-03-29T04:09:38.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fcbbc156cd0 con 0x7fcbbc0cc870 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc0cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43820/0 (socket says 172.21.15.156:43820) 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1872574201 learned_addr learned my addr 172.21.15.156:0/1872574201 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc22d4640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcbbc0cc870 0x7fcbbc1f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc2ad5640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbbc1568f0 0x7fcbbc1ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcbbc0cc870 msgr2=0x7fcbbc1f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcbbc0cc870 0x7fcbbc1f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbbc1568f0 msgr2=0x7fcbbc1ede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbbc1568f0 0x7fcbbc1ede20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 -- 172.21.15.156:0/1872574201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbbc139840 con 0x7fcbbc0cec00 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc2ad5640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbbc1568f0 0x7fcbbc1ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc22d4640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcbbc0cc870 0x7fcbbc1f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:38.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc1ad3640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc0cb7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcbac000c00 tx=0x7fcbac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac032840 con 0x7fcbbc0cec00 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbac032a00 con 0x7fcbbc0cec00 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbbc13bb30 con 0x7fcbbc0cec00 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac032df0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.352+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbbc13c0f0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.353+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbbc068610 con 0x7fcbbc0cec00 2024-03-29T04:09:38.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.354+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcbac04d3f0 con 0x7fcbbc0cec00 2024-03-29T04:09:38.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.354+0000 7fcbb2ffd640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fcba006c530 0x7fcba006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.354+0000 7fcbc22d4640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fcba006c530 0x7fcba006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.354+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcbac02f030 con 0x7fcbbc0cec00 2024-03-29T04:09:38.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.355+0000 7fcbc22d4640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fcba006c530 0x7fcba006e9f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8000f60 tx=0x7fcbb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.358+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fcbac040070 con 0x7fcbbc0cec00 2024-03-29T04:09:38.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.540+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbbc0b8150 con 0x7fcbbc0cec00 2024-03-29T04:09:38.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.541+0000 7fcbb2ffd640 1 -- 172.21.15.156:0/1872574201 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16878 (secure 0 0 0) 0x7fcbac07b840 con 0x7fcbbc0cec00 2024-03-29T04:09:38.521 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:38.521 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":23,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:34.930341+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fcba006c530 msgr2=0x7fcba006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fcba006c530 0x7fcba006e9f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8000f60 tx=0x7fcbb8009040 comp rx=0 tx=0).stop 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 msgr2=0x7fcbbc0cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc0cb7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcbac000c00 tx=0x7fcbac0046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 shutdown_connections 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fcba006c530 0x7fcba006e9f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbbc1568f0 0x7fcbbc1ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbbc0cec00 0x7fcbbc0cb7a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 --2- 172.21.15.156:0/1872574201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcbbc0cc870 0x7fcbbc1f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 >> 172.21.15.156:0/1872574201 conn(0x7fcbbc0b3eb0 msgr2=0x7fcbbc074530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:38.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 shutdown_connections 2024-03-29T04:09:38.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.543+0000 7fcbc4538640 1 -- 172.21.15.156:0/1872574201 wait complete. 2024-03-29T04:09:38.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:38.535 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:38.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.661+0000 7fb31130e640 1 Processor -- start 2024-03-29T04:09:38.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.661+0000 7fb31130e640 1 -- start start 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.662+0000 7fb31130e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 0x7fb30c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.662+0000 7fb31130e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb30c0af6f0 con 0x7fb30c15b7b0 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.662+0000 7fb31130e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb30c0af9d0 con 0x7fb30c0b19e0 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.662+0000 7fb31130e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb30c0b01d0 con 0x7fb30c0b4b70 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.662+0000 7fb303fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb30c15b7b0 0x7fb30c0ae9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33334/0 (socket says 172.21.15.156:33334) 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.662+0000 7fb303fff640 1 -- 172.21.15.156:0/307548246 learned_addr learned my addr 172.21.15.156:0/307548246 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263969510 0 0) 0x7fb30c0af6f0 con 0x7fb30c15b7b0 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2f8003520 con 0x7fb30c15b7b0 2024-03-29T04:09:38.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30affd640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 0x7fb30c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2436728241 0 0) 0x7fb2f8003520 con 0x7fb30c15b7b0 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 msgr2=0x7fb30c0b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 0x7fb30c0b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 >> v1:172.21.15.203:6789/0 conn(0x7fb30c0b19e0 legacy=0x7fb30c165120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c0b5490 con 0x7fb30c15b7b0 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fb2f4001670 con 0x7fb30c15b7b0 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 >> v1:172.21.15.156:6789/0 conn(0x7fb30c15b7b0 legacy=0x7fb30c0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 0x7fb2f80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8003f60 0x7fb2f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f8004d40 0x7fb2f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c0b01d0 con 0x7fb2f8004d40 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f800bb50 con 0x7fb2f8003f60 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f800bd50 con 0x7fb30c0b4b70 2024-03-29T04:09:38.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30b7fe640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f8004d40 0x7fb2f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb303fff640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8003f60 0x7fb2f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30affd640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 0x7fb2f80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30b7fe640 1 -- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 msgr2=0x7fb2f80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30b7fe640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c0b4b70 0x7fb2f80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30b7fe640 1 -- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8003f60 msgr2=0x7fb2f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30b7fe640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8003f60 0x7fb2f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.663+0000 7fb30b7fe640 1 -- 172.21.15.156:0/307548246 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f8003520 con 0x7fb2f8004d40 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb30b7fe640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f8004d40 0x7fb2f80051c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb304001e20 tx=0x7fb2f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896ee879fb07f9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304001a70 con 0x7fb2f8004d40 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb304003b70 con 0x7fb2f8004d40 2024-03-29T04:09:38.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb30a7fc640 1 -- 172.21.15.156:0/307548246 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040050d0 con 0x7fb2f8004d40 2024-03-29T04:09:38.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb31130e640 1 -- 172.21.15.156:0/307548246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f8004d40 msgr2=0x7fb2f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb31130e640 1 --2- 172.21.15.156:0/307548246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f8004d40 0x7fb2f80051c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb304001e20 tx=0x7fb2f8005830 comp rx=0 tx=0).stop 2024-03-29T04:09:38.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb30b7fe640 1 -- 172.21.15.156:0/307548246 reap_dead start 2024-03-29T04:09:38.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb31130e640 1 -- 172.21.15.156:0/307548246 shutdown_connections 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb31130e640 1 -- 172.21.15.156:0/307548246 >> 172.21.15.156:0/307548246 conn(0x7fb30c0b65f0 msgr2=0x7fb30c0b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb31130e640 1 -- 172.21.15.156:0/307548246 shutdown_connections 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.665+0000 7fb31130e640 1 -- 172.21.15.156:0/307548246 wait complete. 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.667+0000 7fb31130e640 1 Processor -- start 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.667+0000 7fb31130e640 1 -- start start 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb31130e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c148d80 0x7fb30c1f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb31130e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 0x7fb30c0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb31130e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c1f6160 0x7fb30c1f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb31130e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c0b52e0 con 0x7fb30c149160 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb31130e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c0b5160 con 0x7fb30c1f6160 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb31130e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb30c0af0d0 con 0x7fb30c148d80 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 0x7fb30c0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 0x7fb30c0c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43850/0 (socket says 172.21.15.156:43850) 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 -- 172.21.15.156:0/2095332888 learned_addr learned my addr 172.21.15.156:0/2095332888 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 -- 172.21.15.156:0/2095332888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c148d80 msgr2=0x7fb30c1f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c148d80 0x7fb30c1f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 -- 172.21.15.156:0/2095332888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c1f6160 msgr2=0x7fb30c1f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c1f6160 0x7fb30c1f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 -- 172.21.15.156:0/2095332888 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c0c6580 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.668+0000 7fb303fff640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 0x7fb30c0c6040 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb2f40072f0 tx=0x7fb2f40077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.669+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f400f8c0 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.669+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f4018070 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.669+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30c0c69d0 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.669+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f400fcc0 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.669+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30c0c6f90 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.670+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30c0c7190 con 0x7fb30c149160 2024-03-29T04:09:38.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.670+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb2f401d460 con 0x7fb30c149160 2024-03-29T04:09:38.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.670+0000 7fb3037fe640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb2ec06c560 0x7fb2ec06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.671+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb2f408b850 con 0x7fb30c149160 2024-03-29T04:09:38.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.671+0000 7fb30affd640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb2ec06c560 0x7fb2ec06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.671+0000 7fb30affd640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb2ec06c560 0x7fb2ec06ea20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009ab0 tx=0x7fb2fc0095d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.675+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb30c0c7190 con 0x7fb30c149160 2024-03-29T04:09:38.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.878+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb30c09f430 con 0x7fb30c149160 2024-03-29T04:09:38.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.878+0000 7fb3037fe640 1 -- 172.21.15.156:0/2095332888 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fb2f400fe60 con 0x7fb30c149160 2024-03-29T04:09:38.858 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:38.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.880+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb2ec06c560 msgr2=0x7fb2ec06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.880+0000 7fb31130e640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb2ec06c560 0x7fb2ec06ea20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009ab0 tx=0x7fb2fc0095d0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.880+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 msgr2=0x7fb30c0c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.880+0000 7fb31130e640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 0x7fb30c0c6040 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb2f40072f0 tx=0x7fb2f40077b0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 shutdown_connections 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb2ec06c560 0x7fb2ec06ea20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c1f6160 0x7fb30c1f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb30c149160 0x7fb30c0c6040 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 --2- 172.21.15.156:0/2095332888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb30c148d80 0x7fb30c1f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 >> 172.21.15.156:0/2095332888 conn(0x7fb30c0b65f0 msgr2=0x7fb30c0b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 shutdown_connections 2024-03-29T04:09:38.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.881+0000 7fb31130e640 1 -- 172.21.15.156:0/2095332888 wait complete. 2024-03-29T04:09:38.871 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:38.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.997+0000 7f8b0b6b7640 1 Processor -- start 2024-03-29T04:09:38.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.997+0000 7f8b0b6b7640 1 -- start start 2024-03-29T04:09:38.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b0b6b7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8b04071020 0x7f8b04071400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b0b6b7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8b04141ca0 0x7f8b0415c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b0b6b7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b04071940 con 0x7f8b0415ceb0 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b0b6b7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8b0415d960 con 0x7f8b04141ca0 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b0b6b7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8b0413f7f0 con 0x7f8b04071020 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b08c52640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8b04141ca0 0x7f8b0415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b09c54640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8b0415ceb0 0x7f8b0415ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33338/0 (socket says 172.21.15.156:33338) 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b09c54640 1 -- 172.21.15.156:0/2134878656 learned_addr learned my addr 172.21.15.156:0/2134878656 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b09453640 1 --2- 172.21.15.156:0/2134878656 >> v2:172.21.15.203:3300/0 conn(0x7f8b04071020 0x7f8b04071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b08c52640 1 -- 172.21.15.156:0/2134878656 >> v2:172.21.15.203:3300/0 conn(0x7f8b04071020 msgr2=0x7f8b04071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b08c52640 1 --2- 172.21.15.156:0/2134878656 >> v2:172.21.15.203:3300/0 conn(0x7f8b04071020 0x7f8b04071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b08c52640 1 -- 172.21.15.156:0/2134878656 >> v1:172.21.15.156:6789/0 conn(0x7f8b0415ceb0 legacy=0x7f8b0415ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8b08c52640 1 -- 172.21.15.156:0/2134878656 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0415d3a0 con 0x7f8b04141ca0 2024-03-29T04:09:38.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.998+0000 7f8afbfff640 1 -- 172.21.15.156:0/2134878656 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678292150 0 0) 0x7f8b04071940 con 0x7f8b0415ceb0 2024-03-29T04:09:38.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.999+0000 7f8b08c52640 1 --2- 172.21.15.156:0/2134878656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b04141ca0 0x7f8b0415c970 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8b0406c560 tx=0x7f8af402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8eb837df8a502 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.999+0000 7f8afbfff640 1 -- 172.21.15.156:0/2134878656 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4035020 con 0x7f8b04141ca0 2024-03-29T04:09:38.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.999+0000 7f8afbfff640 1 -- 172.21.15.156:0/2134878656 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af4031830 con 0x7f8b04141ca0 2024-03-29T04:09:38.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:38.999+0000 7f8afbfff640 1 -- 172.21.15.156:0/2134878656 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4031c00 con 0x7f8b04141ca0 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/2134878656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b04141ca0 msgr2=0x7f8b0415c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/2134878656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b04141ca0 0x7f8b0415c970 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8b0406c560 tx=0x7f8af402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/2134878656 shutdown_connections 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/2134878656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b04141ca0 0x7f8b0415c970 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/2134878656 >> v2:172.21.15.203:3300/0 conn(0x7f8b04071020 0x7f8b04071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/2134878656 >> 172.21.15.156:0/2134878656 conn(0x7f8b040b3eb0 msgr2=0x7f8b040b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:38.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/2134878656 shutdown_connections 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.000+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/2134878656 wait complete. 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.001+0000 7f8b0b6b7640 1 Processor -- start 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.001+0000 7f8b0b6b7640 1 -- start start 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8b04071020 0x7f8b04136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b04141ca0 0x7f8b041447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 0x7f8b04144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8b040d1280 con 0x7f8b0415ceb0 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8b040d0fa0 con 0x7f8b04141ca0 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8b040d27b0 con 0x7f8b04071020 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 0x7f8b04144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 0x7f8b04144ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43860/0 (socket says 172.21.15.156:43860) 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 -- 172.21.15.156:0/912556536 learned_addr learned my addr 172.21.15.156:0/912556536 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:38.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09453640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8b04071020 0x7f8b04136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b08c52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b04141ca0 0x7f8b041447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 -- 172.21.15.156:0/912556536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8b04071020 msgr2=0x7f8b04136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8b04071020 0x7f8b04136b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 -- 172.21.15.156:0/912556536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b04141ca0 msgr2=0x7f8b041447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b04141ca0 0x7f8b041447a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 -- 172.21.15.156:0/912556536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0406e6c0 con 0x7f8b0415ceb0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b09c54640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 0x7f8b04144ce0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c780 tx=0x7f8afc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc018070 con 0x7f8b0415ceb0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc00ce30 con 0x7f8b0415ceb0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b04137080 con 0x7f8b0415ceb0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.002+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00a840 con 0x7f8b0415ceb0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.003+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b0406d900 con 0x7f8b0415ceb0 2024-03-29T04:09:38.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.004+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b04070e10 con 0x7f8b0415ceb0 2024-03-29T04:09:38.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.004+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8afc01b070 con 0x7f8b0415ceb0 2024-03-29T04:09:38.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.004+0000 7f8af9ffb640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ae806c530 0x7f8ae806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:38.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.005+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8afc08ab90 con 0x7f8b0415ceb0 2024-03-29T04:09:38.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.005+0000 7f8b09453640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ae806c530 0x7f8ae806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:38.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.005+0000 7f8b09453640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ae806c530 0x7f8ae806e9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8b00006fd0 tx=0x7f8b00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:38.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.009+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8afc058ae0 con 0x7f8b0415ceb0 2024-03-29T04:09:39.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.189+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b04071ce0 con 0x7f8b0415ceb0 2024-03-29T04:09:39.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.189+0000 7f8af9ffb640 1 -- 172.21.15.156:0/912556536 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f8afc05cde0 con 0x7f8b0415ceb0 2024-03-29T04:09:39.169 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:39.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ae806c530 msgr2=0x7f8ae806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ae806c530 0x7f8ae806e9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8b00006fd0 tx=0x7f8b00008040 comp rx=0 tx=0).stop 2024-03-29T04:09:39.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 msgr2=0x7f8b04144ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 0x7f8b04144ce0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c780 tx=0x7f8afc00cc40 comp rx=0 tx=0).stop 2024-03-29T04:09:39.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 shutdown_connections 2024-03-29T04:09:39.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ae806c530 0x7f8ae806e9f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b0415ceb0 0x7f8b04144ce0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b04141ca0 0x7f8b041447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 --2- 172.21.15.156:0/912556536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8b04071020 0x7f8b04136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 >> 172.21.15.156:0/912556536 conn(0x7f8b040b3eb0 msgr2=0x7f8b0415e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:39.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 shutdown_connections 2024-03-29T04:09:39.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.191+0000 7f8b0b6b7640 1 -- 172.21.15.156:0/912556536 wait complete. 2024-03-29T04:09:39.182 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:39.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.338+0000 7fa978e70640 1 Processor -- start 2024-03-29T04:09:39.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.338+0000 7fa978e70640 1 -- start start 2024-03-29T04:09:39.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa978e70640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 0x7fa9740d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa978e70640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 0x7fa9740d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa978e70640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9740b30f0 con 0x7fa97415c330 2024-03-29T04:09:39.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa978e70640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa9740b33d0 con 0x7fa9740b3eb0 2024-03-29T04:09:39.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa978e70640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa9740b36b0 con 0x7fa9740cf9a0 2024-03-29T04:09:39.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa972d9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 0x7fa9740d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa971d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 0x7fa9740d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa971d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 0x7fa9740d0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:33130/0 (socket says 172.21.15.156:33130) 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa972d9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 0x7fa9740d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:57586/0 (socket says 172.21.15.156:57586) 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.339+0000 7fa971d9b640 1 -- 172.21.15.156:0/2388088086 learned_addr learned my addr 172.21.15.156:0/2388088086 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.340+0000 7fa972d9d640 1 -- 172.21.15.156:0/2388088086 >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 msgr2=0x7fa9740d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.340+0000 7fa972d9d640 1 --2- 172.21.15.156:0/2388088086 >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 0x7fa9740d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.340+0000 7fa972d9d640 1 -- 172.21.15.156:0/2388088086 >> v1:172.21.15.203:6789/0 conn(0x7fa97415c330 legacy=0x7fa9740ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.340+0000 7fa972d9d640 1 -- 172.21.15.156:0/2388088086 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9740b3990 con 0x7fa9740cf9a0 2024-03-29T04:09:39.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.340+0000 7fa971d9b640 1 --2- 172.21.15.156:0/2388088086 >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 0x7fa9740d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.340+0000 7fa972d9d640 1 --2- 172.21.15.156:0/2388088086 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 0x7fa9740d07c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9640097a0 tx=0x7fa96402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c179e2294bf9b1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa97159a640 1 -- 172.21.15.156:0/2388088086 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964035020 con 0x7fa9740cf9a0 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa97159a640 1 -- 172.21.15.156:0/2388088086 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa964031830 con 0x7fa9740cf9a0 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa97159a640 1 -- 172.21.15.156:0/2388088086 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964031c00 con 0x7fa9740cf9a0 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 -- 172.21.15.156:0/2388088086 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 msgr2=0x7fa9740d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 --2- 172.21.15.156:0/2388088086 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 0x7fa9740d07c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9640097a0 tx=0x7fa96402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 -- 172.21.15.156:0/2388088086 shutdown_connections 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 --2- 172.21.15.156:0/2388088086 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9740cf9a0 0x7fa9740d07c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 --2- 172.21.15.156:0/2388088086 >> v2:172.21.15.203:3300/0 conn(0x7fa9740b3eb0 0x7fa9740d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 -- 172.21.15.156:0/2388088086 >> 172.21.15.156:0/2388088086 conn(0x7fa9740b1f30 msgr2=0x7fa9740b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 -- 172.21.15.156:0/2388088086 shutdown_connections 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.341+0000 7fa978e70640 1 -- 172.21.15.156:0/2388088086 wait complete. 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 Processor -- start 2024-03-29T04:09:39.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 -- start start 2024-03-29T04:09:39.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 0x7fa974155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9740cf9a0 0x7fa974153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa97415c330 0x7fa974151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa97406db30 con 0x7fa9740b3eb0 2024-03-29T04:09:39.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9740ad880 con 0x7fa9740cf9a0 2024-03-29T04:09:39.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa978e70640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa97406dcb0 con 0x7fa97415c330 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa97259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 0x7fa974155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa97259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 0x7fa974155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43874/0 (socket says 172.21.15.156:43874) 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.343+0000 7fa97259c640 1 -- 172.21.15.156:0/2830534610 learned_addr learned my addr 172.21.15.156:0/2830534610 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa971d9b640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9740cf9a0 0x7fa974153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa972d9d640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa97415c330 0x7fa974151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa97259c640 1 -- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa97415c330 msgr2=0x7fa974151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa97259c640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa97415c330 0x7fa974151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa97259c640 1 -- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9740cf9a0 msgr2=0x7fa974153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa97259c640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9740cf9a0 0x7fa974153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa97259c640 1 -- 172.21.15.156:0/2830534610 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa974153f80 con 0x7fa9740b3eb0 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa971d9b640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9740cf9a0 0x7fa974153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa97259c640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 0x7fa974155b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00c6f0 tx=0x7fa96c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c003990 con 0x7fa9740b3eb0 2024-03-29T04:09:39.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa96c003b50 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa974154260 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c00a850 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.344+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa974154820 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.345+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa96c00a9f0 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.345+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9741f48c0 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.346+0000 7fa96affd640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa95406c530 0x7fa95406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.346+0000 7fa971d9b640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa95406c530 0x7fa95406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.346+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa96c08b680 con 0x7fa9740b3eb0 2024-03-29T04:09:39.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.346+0000 7fa971d9b640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa95406c530 0x7fa95406e9f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009630 tx=0x7fa95c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:39.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.354+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa96c059b90 con 0x7fa9740b3eb0 2024-03-29T04:09:39.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.660+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa97404f9d0 con 0x7fa9740b3eb0 2024-03-29T04:09:39.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.661+0000 7fa96affd640 1 -- 172.21.15.156:0/2830534610 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fa96c059490 con 0x7fa9740b3eb0 2024-03-29T04:09:39.640 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:39.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa95406c530 msgr2=0x7fa95406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa95406c530 0x7fa95406e9f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009630 tx=0x7fa95c008040 comp rx=0 tx=0).stop 2024-03-29T04:09:39.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 msgr2=0x7fa974155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 0x7fa974155b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00c6f0 tx=0x7fa96c00cbb0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 shutdown_connections 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa95406c530 0x7fa95406e9f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa97415c330 0x7fa974151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9740cf9a0 0x7fa974153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.663+0000 7fa978e70640 1 --2- 172.21.15.156:0/2830534610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9740b3eb0 0x7fa974155b70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.664+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 >> 172.21.15.156:0/2830534610 conn(0x7fa9740b1f30 msgr2=0x7fa97406e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.664+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 shutdown_connections 2024-03-29T04:09:39.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.664+0000 7fa978e70640 1 -- 172.21.15.156:0/2830534610 wait complete. 2024-03-29T04:09:39.657 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:09:39.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.809+0000 7f0da19f2640 1 Processor -- start 2024-03-29T04:09:39.789 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.809+0000 7f0da19f2640 1 -- start start 2024-03-29T04:09:39.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0da19f2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0d9c0d2120 0x7f0d9c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0da19f2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0d9c06cb10 0x7f0d9c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0da19f2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9c0d2df0 con 0x7f0d9c15e920 2024-03-29T04:09:39.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0da19f2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9c0cf7a0 con 0x7f0d9c0d2120 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0da19f2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9c0cf190 con 0x7f0d9c06cb10 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0d9a7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0d9c0d2120 0x7f0d9c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0d9affd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f0d9c15e920 0x7f0d9c0d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33350/0 (socket says 172.21.15.156:33350) 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0d9affd640 1 -- 172.21.15.156:0/3731757809 learned_addr learned my addr 172.21.15.156:0/3731757809 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.810+0000 7f0d9b7fe640 1 --2- 172.21.15.156:0/3731757809 >> v2:172.21.15.203:3300/0 conn(0x7f0d9c06cb10 0x7f0d9c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.811+0000 7f0d9a7fc640 1 -- 172.21.15.156:0/3731757809 >> v2:172.21.15.203:3300/0 conn(0x7f0d9c06cb10 msgr2=0x7f0d9c0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.811+0000 7f0d9a7fc640 1 --2- 172.21.15.156:0/3731757809 >> v2:172.21.15.203:3300/0 conn(0x7f0d9c06cb10 0x7f0d9c0cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.811+0000 7f0d9a7fc640 1 -- 172.21.15.156:0/3731757809 >> v1:172.21.15.156:6789/0 conn(0x7f0d9c15e920 legacy=0x7f0d9c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.811+0000 7f0d99ffb640 1 -- 172.21.15.156:0/3731757809 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048316748 0 0) 0x7f0d9c0d2df0 con 0x7f0d9c15e920 2024-03-29T04:09:39.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.811+0000 7f0d9a7fc640 1 -- 172.21.15.156:0/3731757809 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c0d3080 con 0x7f0d9c0d2120 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.811+0000 7f0d9a7fc640 1 --2- 172.21.15.156:0/3731757809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c0d2120 0x7f0d9c06c5d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c08ca00 tx=0x7f0d8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c987d3f442bc7fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0d99ffb640 1 -- 172.21.15.156:0/3731757809 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c035020 con 0x7f0d9c0d2120 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0d99ffb640 1 -- 172.21.15.156:0/3731757809 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8c031830 con 0x7f0d9c0d2120 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0d99ffb640 1 -- 172.21.15.156:0/3731757809 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c031c00 con 0x7f0d9c0d2120 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 -- 172.21.15.156:0/3731757809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c0d2120 msgr2=0x7f0d9c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3731757809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c0d2120 0x7f0d9c06c5d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c08ca00 tx=0x7f0d8c02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 -- 172.21.15.156:0/3731757809 shutdown_connections 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3731757809 >> v2:172.21.15.203:3300/0 conn(0x7f0d9c06cb10 0x7f0d9c0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3731757809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c0d2120 0x7f0d9c06c5d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 -- 172.21.15.156:0/3731757809 >> 172.21.15.156:0/3731757809 conn(0x7f0d9c09d7a0 msgr2=0x7f0d9c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:39.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 -- 172.21.15.156:0/3731757809 shutdown_connections 2024-03-29T04:09:39.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.812+0000 7f0da19f2640 1 -- 172.21.15.156:0/3731757809 wait complete. 2024-03-29T04:09:39.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.813+0000 7f0da19f2640 1 Processor -- start 2024-03-29T04:09:39.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 -- start start 2024-03-29T04:09:39.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 0x7f0d9c0cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d9c0d2120 0x7f0d9c0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d9c15e920 0x7f0d9c15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c0d34e0 con 0x7f0d9c06cb10 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c0d3360 con 0x7f0d9c15e920 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0da19f2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9c0d0170 con 0x7f0d9c0d2120 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0d9affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 0x7f0d9c0cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0d9affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 0x7f0d9c0cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43894/0 (socket says 172.21.15.156:43894) 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.814+0000 7f0d9affd640 1 -- 172.21.15.156:0/3437809890 learned_addr learned my addr 172.21.15.156:0/3437809890 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9b7fe640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d9c15e920 0x7f0d9c15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9a7fc640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d9c0d2120 0x7f0d9c0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9affd640 1 -- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d9c0d2120 msgr2=0x7f0d9c0b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9affd640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d9c0d2120 0x7f0d9c0b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9affd640 1 -- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d9c15e920 msgr2=0x7f0d9c15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9affd640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d9c15e920 0x7f0d9c15a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9affd640 1 -- 172.21.15.156:0/3437809890 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c0b5880 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9b7fe640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d9c15e920 0x7f0d9c15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d9affd640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 0x7f0d9c0cb790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c460 tx=0x7f0d8400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84018070 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d84007500 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9c0b5b60 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84004a20 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.815+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9c0b6120 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.816+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d9c0a2480 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.817+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0d8401b070 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.817+0000 7f0d93fff640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0d7c06c530 0x7f0d7c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.817+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0d8408a980 con 0x7f0d9c06cb10 2024-03-29T04:09:39.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.817+0000 7f0d9a7fc640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0d7c06c530 0x7f0d7c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:39.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.818+0000 7f0d9a7fc640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0d7c06c530 0x7f0d7c06e9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c036000 tx=0x7f0d8c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:39.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:39.822+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0d84058fb0 con 0x7f0d9c06cb10 2024-03-29T04:09:39.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.016+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0d9c04f9d0 con 0x7f0d9c06cb10 2024-03-29T04:09:39.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.016+0000 7f0d93fff640 1 -- 172.21.15.156:0/3437809890 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f0d840588b0 con 0x7f0d9c06cb10 2024-03-29T04:09:39.996 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:09:39.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0d7c06c530 msgr2=0x7f0d7c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0d7c06c530 0x7f0d7c06e9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c036000 tx=0x7f0d8c008030 comp rx=0 tx=0).stop 2024-03-29T04:09:39.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 msgr2=0x7f0d9c0cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:39.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 0x7f0d9c0cb790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c460 tx=0x7f0d8400c920 comp rx=0 tx=0).stop 2024-03-29T04:09:39.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 shutdown_connections 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0d7c06c530 0x7f0d7c06e9f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d9c15e920 0x7f0d9c15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d9c0d2120 0x7f0d9c0b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 --2- 172.21.15.156:0/3437809890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9c06cb10 0x7f0d9c0cb790 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.018+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 >> 172.21.15.156:0/3437809890 conn(0x7f0d9c09d7a0 msgr2=0x7f0d9c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.019+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 shutdown_connections 2024-03-29T04:09:39.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.019+0000 7f0da19f2640 1 -- 172.21.15.156:0/3437809890 wait complete. 2024-03-29T04:09:40.009 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi203.front.sepia.ceph.com... 2024-03-29T04:09:40.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:40.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-29T04:09:40.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:40.042 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:09:40.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:09:40.042 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-29T04:09:40.099 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T04:09:40.100 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:40.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.223+0000 7f0f0df50640 1 Processor -- start 2024-03-29T04:09:40.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.223+0000 7f0f0df50640 1 -- start start 2024-03-29T04:09:40.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.224+0000 7f0f0df50640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0f0815e730 0x7f0f080ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.224+0000 7f0f0df50640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0f080cf3e0 0x7f0f080d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.224+0000 7f0f0df50640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f080cfa20 con 0x7f0f080d4800 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.224+0000 7f0f0df50640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f0f0806c720 con 0x7f0f0815e730 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.224+0000 7f0f0df50640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0f080cfd00 con 0x7f0f080cf3e0 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f077fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0f0815e730 0x7f0f080ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f07fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f0f080d4800 0x7f0f080d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33364/0 (socket says 172.21.15.156:33364) 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f07fff640 1 -- 172.21.15.156:0/4203067774 learned_addr learned my addr 172.21.15.156:0/4203067774 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f06ffd640 1 --2- 172.21.15.156:0/4203067774 >> v2:172.21.15.203:3300/0 conn(0x7f0f080cf3e0 0x7f0f080d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f077fe640 1 -- 172.21.15.156:0/4203067774 >> v2:172.21.15.203:3300/0 conn(0x7f0f080cf3e0 msgr2=0x7f0f080d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f077fe640 1 --2- 172.21.15.156:0/4203067774 >> v2:172.21.15.203:3300/0 conn(0x7f0f080cf3e0 0x7f0f080d42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f077fe640 1 -- 172.21.15.156:0/4203067774 >> v1:172.21.15.156:6789/0 conn(0x7f0f080d4800 legacy=0x7f0f080d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f077fe640 1 -- 172.21.15.156:0/4203067774 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f080cffe0 con 0x7f0f0815e730 2024-03-29T04:09:40.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.225+0000 7f0f067fc640 1 -- 172.21.15.156:0/4203067774 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245751519 0 0) 0x7f0f080cfa20 con 0x7f0f080d4800 2024-03-29T04:09:40.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f077fe640 1 --2- 172.21.15.156:0/4203067774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f0815e730 0x7f0f080ceea0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ba4e0 tx=0x7f0ef002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957a8156659b0b60 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f067fc640 1 -- 172.21.15.156:0/4203067774 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0035020 con 0x7f0f0815e730 2024-03-29T04:09:40.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f067fc640 1 -- 172.21.15.156:0/4203067774 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef0031830 con 0x7f0f0815e730 2024-03-29T04:09:40.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f067fc640 1 -- 172.21.15.156:0/4203067774 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0031c00 con 0x7f0f0815e730 2024-03-29T04:09:40.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 -- 172.21.15.156:0/4203067774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f0815e730 msgr2=0x7f0f080ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 --2- 172.21.15.156:0/4203067774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f0815e730 0x7f0f080ceea0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ba4e0 tx=0x7f0ef002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 -- 172.21.15.156:0/4203067774 shutdown_connections 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 --2- 172.21.15.156:0/4203067774 >> v2:172.21.15.203:3300/0 conn(0x7f0f080cf3e0 0x7f0f080d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 --2- 172.21.15.156:0/4203067774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f0815e730 0x7f0f080ceea0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 -- 172.21.15.156:0/4203067774 >> 172.21.15.156:0/4203067774 conn(0x7f0f0808bfd0 msgr2=0x7f0f0809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 -- 172.21.15.156:0/4203067774 shutdown_connections 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.226+0000 7f0f0df50640 1 -- 172.21.15.156:0/4203067774 wait complete. 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.227+0000 7f0f0df50640 1 Processor -- start 2024-03-29T04:09:40.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 -- start start 2024-03-29T04:09:40.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 0x7f0f08159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0f080d4800 0x7f0f08153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f0815e730 0x7f0f081583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0f080d2220 con 0x7f0f080cf3e0 2024-03-29T04:09:40.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f080d4c30 con 0x7f0f0815e730 2024-03-29T04:09:40.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f0df50640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0815eb10 con 0x7f0f080d4800 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.228+0000 7f0f077fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 0x7f0f08159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 0x7f0f08159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43926/0 (socket says 172.21.15.156:43926) 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 -- 172.21.15.156:0/184296644 learned_addr learned my addr 172.21.15.156:0/184296644 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f07fff640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f0815e730 0x7f0f081583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f06ffd640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0f080d4800 0x7f0f08153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 -- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0f080d4800 msgr2=0x7f0f08153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0f080d4800 0x7f0f08153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 -- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f0815e730 msgr2=0x7f0f081583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f0815e730 0x7f0f081583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 -- 172.21.15.156:0/184296644 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f080c8a70 con 0x7f0f080cf3e0 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f06ffd640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0f080d4800 0x7f0f08153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f07fff640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f0815e730 0x7f0f081583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f077fe640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 0x7f0f08159eb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0035040 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef0004230 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef003a640 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0815a570 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.229+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0815ab30 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.230+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f08068610 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.231+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0ef0038020 con 0x7f0f080cf3e0 2024-03-29T04:09:40.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.231+0000 7f0efffff640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0ee806c560 0x7f0ee806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.231+0000 7f0f06ffd640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0ee806c560 0x7f0ee806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.231+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0ef00ad3f0 con 0x7f0f080cf3e0 2024-03-29T04:09:40.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.232+0000 7f0f06ffd640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0ee806c560 0x7f0ee806ea20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ce9e0 tx=0x7f0ef8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.236+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0ef007ba20 con 0x7f0f080cf3e0 2024-03-29T04:09:40.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.455+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f0806da10 con 0x7f0f080cf3e0 2024-03-29T04:09:40.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.456+0000 7f0efffff640 1 -- 172.21.15.156:0/184296644 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f0ef007b320 con 0x7f0f080cf3e0 2024-03-29T04:09:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:40.436 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:40.436 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:40.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.457+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0ee806c560 msgr2=0x7f0ee806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0ee806c560 0x7f0ee806ea20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ce9e0 tx=0x7f0ef8008040 comp rx=0 tx=0).stop 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 msgr2=0x7f0f08159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 0x7f0f08159eb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef00046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 shutdown_connections 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0ee806c560 0x7f0ee806ea20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f0815e730 0x7f0f081583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0f080d4800 0x7f0f08153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 --2- 172.21.15.156:0/184296644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f080cf3e0 0x7f0f08159eb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 >> 172.21.15.156:0/184296644 conn(0x7f0f0808bfd0 msgr2=0x7f0f080d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 shutdown_connections 2024-03-29T04:09:40.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.458+0000 7f0f0df50640 1 -- 172.21.15.156:0/184296644 wait complete. 2024-03-29T04:09:40.448 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:40.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.570+0000 7f2d4bdaa640 1 Processor -- start 2024-03-29T04:09:40.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.570+0000 7f2d4bdaa640 1 -- start start 2024-03-29T04:09:40.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d4bdaa640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 0x7f2d440d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d4bdaa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2d4406cbb0 0x7f2d4406d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d4bdaa640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d440d4c80 con 0x7f2d4415e730 2024-03-29T04:09:40.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d4bdaa640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4406c720 con 0x7f2d4406cbb0 2024-03-29T04:09:40.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d4bdaa640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2d440cfa20 con 0x7f2d440d42c0 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d49b46640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2d4415e730 0x7f2d440ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33378/0 (socket says 172.21.15.156:33378) 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d49b46640 1 -- 172.21.15.156:0/914622316 learned_addr learned my addr 172.21.15.156:0/914622316 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d4a347640 1 --2- 172.21.15.156:0/914622316 >> v2:172.21.15.203:3300/0 conn(0x7f2d4406cbb0 0x7f2d4406d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d49345640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 0x7f2d440d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.571+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789684054 0 0) 0x7f2d440d4c80 con 0x7f2d4415e730 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d30003520 con 0x7f2d4415e730 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1435336494 0 0) 0x7f2d30003520 con 0x7f2d4415e730 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 msgr2=0x7f2d440d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 0x7f2d440d4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 >> v2:172.21.15.203:3300/0 conn(0x7f2d4406cbb0 msgr2=0x7f2d4406d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:40.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 --2- 172.21.15.156:0/914622316 >> v2:172.21.15.203:3300/0 conn(0x7f2d4406cbb0 0x7f2d4406d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d440cfd00 con 0x7f2d4415e730 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f2d40001670 con 0x7f2d4415e730 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 >> v1:172.21.15.156:6789/0 conn(0x7f2d4415e730 legacy=0x7f2d440ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d300039b0 0x7f2d30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30004800 0x7f2d30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 0x7f2d300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4406c720 con 0x7f2d300039b0 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440cfa20 con 0x7f2d30004800 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3000b9c0 con 0x7f2d440d42c0 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49b46640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d300039b0 0x7f2d30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49345640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30004800 0x7f2d30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d4a347640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 0x7f2d300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49b46640 1 -- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 msgr2=0x7f2d300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49b46640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d440d42c0 0x7f2d300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49b46640 1 -- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30004800 msgr2=0x7f2d30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49b46640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30004800 0x7f2d30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49b46640 1 -- 172.21.15.156:0/914622316 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d30003520 con 0x7f2d300039b0 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.572+0000 7f2d49345640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d30004800 0x7f2d30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.573+0000 7f2d49b46640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d300039b0 0x7f2d30003dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2d40001050 tx=0x7f2d30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9009d97aad4ced6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.573+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40007d70 con 0x7f2d300039b0 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.573+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d40030070 con 0x7f2d300039b0 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.573+0000 7f2d48b44640 1 -- 172.21.15.156:0/914622316 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4002cd80 con 0x7f2d300039b0 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/914622316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d300039b0 msgr2=0x7f2d30003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/914622316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d300039b0 0x7f2d30003dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2d40001050 tx=0x7f2d30005830 comp rx=0 tx=0).stop 2024-03-29T04:09:40.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4a347640 1 -- 172.21.15.156:0/914622316 reap_dead start 2024-03-29T04:09:40.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/914622316 shutdown_connections 2024-03-29T04:09:40.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/914622316 >> 172.21.15.156:0/914622316 conn(0x7f2d4408bfd0 msgr2=0x7f2d4409d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:40.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/914622316 shutdown_connections 2024-03-29T04:09:40.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.574+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/914622316 wait complete. 2024-03-29T04:09:40.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.575+0000 7f2d4bdaa640 1 Processor -- start 2024-03-29T04:09:40.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.576+0000 7f2d4bdaa640 1 -- start start 2024-03-29T04:09:40.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d300039b0 0x7f2d440ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 0x7f2d440b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d441f4130 0x7f2d441f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440d2220 con 0x7f2d440b4a10 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440d02c0 con 0x7f2d300039b0 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4415eb10 con 0x7f2d441f4130 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 0x7f2d440b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 0x7f2d440b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43954/0 (socket says 172.21.15.156:43954) 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 -- 172.21.15.156:0/3175506067 learned_addr learned my addr 172.21.15.156:0/3175506067 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49b46640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d300039b0 0x7f2d440ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4a347640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d441f4130 0x7f2d441f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 -- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d441f4130 msgr2=0x7f2d441f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d441f4130 0x7f2d441f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 -- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d300039b0 msgr2=0x7f2d440ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d300039b0 0x7f2d440ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 -- 172.21.15.156:0/3175506067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d440cac60 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49b46640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d300039b0 0x7f2d440ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4a347640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d441f4130 0x7f2d441f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d49345640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 0x7f2d440b4e90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2d340090e0 tx=0x7f2d34011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34019070 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d34005e00 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d441f4ed0 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34004260 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.577+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d440cc8d0 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.579+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d44068610 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.579+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2d34003610 con 0x7f2d440b4a10 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.579+0000 7f2d3a7fc640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d2806c560 0x7f2d2806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.579+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2d3408bb10 con 0x7f2d440b4a10 2024-03-29T04:09:40.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.579+0000 7f2d49b46640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d2806c560 0x7f2d2806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.580+0000 7f2d49b46640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d2806c560 0x7f2d2806ea20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d40034000 tx=0x7f2d40007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.584+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2d3405a0b0 con 0x7f2d440b4a10 2024-03-29T04:09:40.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.766+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d440ba2c0 con 0x7f2d440b4a10 2024-03-29T04:09:40.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.767+0000 7f2d3a7fc640 1 -- 172.21.15.156:0/3175506067 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16878 (secure 0 0 0) 0x7f2d340599b0 con 0x7f2d440b4a10 2024-03-29T04:09:40.747 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:40.747 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":23,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:34.930341+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:40.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d2806c560 msgr2=0x7f2d2806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d2806c560 0x7f2d2806ea20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d40034000 tx=0x7f2d40007000 comp rx=0 tx=0).stop 2024-03-29T04:09:40.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 msgr2=0x7f2d440b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 0x7f2d440b4e90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2d340090e0 tx=0x7f2d34011a60 comp rx=0 tx=0).stop 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 shutdown_connections 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2d2806c560 0x7f2d2806ea20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2d441f4130 0x7f2d441f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d440b4a10 0x7f2d440b4e90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 --2- 172.21.15.156:0/3175506067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d300039b0 0x7f2d440ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 >> 172.21.15.156:0/3175506067 conn(0x7f2d4408bfd0 msgr2=0x7f2d440b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.770+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 shutdown_connections 2024-03-29T04:09:40.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.771+0000 7f2d4bdaa640 1 -- 172.21.15.156:0/3175506067 wait complete. 2024-03-29T04:09:40.761 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:08:51.621554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:08:52.104255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:40.762 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:40.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.890+0000 7fa60ea88640 1 Processor -- start 2024-03-29T04:09:40.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.890+0000 7fa60ea88640 1 -- start start 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.890+0000 7fa60ea88640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa608145c70 0x7fa6080b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.890+0000 7fa60ea88640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa6080b4310 0x7fa6080cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60ea88640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60815cb10 con 0x7fa6080cf340 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60ea88640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa60815cdf0 con 0x7fa608145c70 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60ea88640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa60815d0d0 con 0x7fa6080b4310 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60c824640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa608145c70 0x7fa6080b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60d025640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa6080cf340 0x7fa60806f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:33382/0 (socket says 172.21.15.156:33382) 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60d025640 1 -- 172.21.15.156:0/1283568528 learned_addr learned my addr 172.21.15.156:0/1283568528 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa607fff640 1 --2- 172.21.15.156:0/1283568528 >> v2:172.21.15.203:3300/0 conn(0x7fa6080b4310 0x7fa6080cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60c824640 1 -- 172.21.15.156:0/1283568528 >> v2:172.21.15.203:3300/0 conn(0x7fa6080b4310 msgr2=0x7fa6080cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60c824640 1 --2- 172.21.15.156:0/1283568528 >> v2:172.21.15.203:3300/0 conn(0x7fa6080b4310 0x7fa6080cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60c824640 1 -- 172.21.15.156:0/1283568528 >> v1:172.21.15.156:6789/0 conn(0x7fa6080cf340 legacy=0x7fa60806f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa6077fe640 1 -- 172.21.15.156:0/1283568528 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1212753147 0 0) 0x7fa60815cb10 con 0x7fa6080cf340 2024-03-29T04:09:40.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.891+0000 7fa60c824640 1 -- 172.21.15.156:0/1283568528 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60815d3b0 con 0x7fa608145c70 2024-03-29T04:09:40.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.892+0000 7fa60c824640 1 --2- 172.21.15.156:0/1283568528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa608145c70 0x7fa6080b3dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa6080b5620 tx=0x7fa5fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21310105abb0c16f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.892+0000 7fa6077fe640 1 -- 172.21.15.156:0/1283568528 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0315a0 con 0x7fa608145c70 2024-03-29T04:09:40.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.892+0000 7fa6077fe640 1 -- 172.21.15.156:0/1283568528 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5fc037070 con 0x7fa608145c70 2024-03-29T04:09:40.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.892+0000 7fa6077fe640 1 -- 172.21.15.156:0/1283568528 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc031960 con 0x7fa608145c70 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 -- 172.21.15.156:0/1283568528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa608145c70 msgr2=0x7fa6080b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1283568528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa608145c70 0x7fa6080b3dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa6080b5620 tx=0x7fa5fc02b840 comp rx=0 tx=0).stop 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 -- 172.21.15.156:0/1283568528 shutdown_connections 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1283568528 >> v2:172.21.15.203:3300/0 conn(0x7fa6080b4310 0x7fa6080cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1283568528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa608145c70 0x7fa6080b3dd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 -- 172.21.15.156:0/1283568528 >> 172.21.15.156:0/1283568528 conn(0x7fa6080b7660 msgr2=0x7fa6080b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 -- 172.21.15.156:0/1283568528 shutdown_connections 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.893+0000 7fa60ea88640 1 -- 172.21.15.156:0/1283568528 wait complete. 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.894+0000 7fa60ea88640 1 Processor -- start 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.894+0000 7fa60ea88640 1 -- start start 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60ea88640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6080b4310 0x7fa608141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60ea88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 0x7fa6080ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60ea88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa608145c70 0x7fa608143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60ea88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6081461d0 con 0x7fa6080cf340 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60ea88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa608146050 con 0x7fa608145c70 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60ea88640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa6080d1f00 con 0x7fa6080b4310 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 0x7fa6080ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 0x7fa6080ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43976/0 (socket says 172.21.15.156:43976) 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 -- 172.21.15.156:0/1771168698 learned_addr learned my addr 172.21.15.156:0/1771168698 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60c824640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6080b4310 0x7fa608141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa60d025640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa608145c70 0x7fa608143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 -- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6080b4310 msgr2=0x7fa608141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6080b4310 0x7fa608141ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 -- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa608145c70 msgr2=0x7fa608143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa608145c70 0x7fa608143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 -- 172.21.15.156:0/1771168698 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6080c7540 con 0x7fa6080cf340 2024-03-29T04:09:40.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.895+0000 7fa607fff640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 0x7fa6080ce150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400d800 tx=0x7fa5f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.896+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400b6c0 con 0x7fa6080cf340 2024-03-29T04:09:40.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.896+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f4019070 con 0x7fa6080cf340 2024-03-29T04:09:40.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.896+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6080c7820 con 0x7fa6080cf340 2024-03-29T04:09:40.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.896+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400ba80 con 0x7fa6080cf340 2024-03-29T04:09:40.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.896+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6080c7de0 con 0x7fa6080cf340 2024-03-29T04:09:40.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.897+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6080c8120 con 0x7fa6080cf340 2024-03-29T04:09:40.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.897+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa5f400bc20 con 0x7fa6080cf340 2024-03-29T04:09:40.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.897+0000 7fa6057fa640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5ec06c530 0x7fa5ec06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:40.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.898+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa5f408bc30 con 0x7fa6080cf340 2024-03-29T04:09:40.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.898+0000 7fa60c824640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5ec06c530 0x7fa5ec06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:40.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.898+0000 7fa60c824640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5ec06c530 0x7fa5ec06e9f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc036000 tx=0x7fa5fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:40.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:40.902+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa6080c8120 con 0x7fa6080cf340 2024-03-29T04:09:41.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.083+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa6080d2f60 con 0x7fa6080cf340 2024-03-29T04:09:41.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.084+0000 7fa6057fa640 1 -- 172.21.15.156:0/1771168698 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fa5f401c360 con 0x7fa6080cf340 2024-03-29T04:09:41.064 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5ec06c530 msgr2=0x7fa5ec06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5ec06c530 0x7fa5ec06e9f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc036000 tx=0x7fa5fc008030 comp rx=0 tx=0).stop 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 msgr2=0x7fa6080ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 0x7fa6080ce150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400d800 tx=0x7fa5f400dcc0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 shutdown_connections 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa5ec06c530 0x7fa5ec06e9f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa608145c70 0x7fa608143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6080cf340 0x7fa6080ce150 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 --2- 172.21.15.156:0/1771168698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6080b4310 0x7fa608141ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 >> 172.21.15.156:0/1771168698 conn(0x7fa6080b7660 msgr2=0x7fa608160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:41.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 shutdown_connections 2024-03-29T04:09:41.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.086+0000 7fa60ea88640 1 -- 172.21.15.156:0/1771168698 wait complete. 2024-03-29T04:09:41.077 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:09:41.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.202+0000 7fa04b140640 1 Processor -- start 2024-03-29T04:09:41.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.202+0000 7fa04b140640 1 -- start start 2024-03-29T04:09:41.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa04b140640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0440d2120 0x7fa04406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa04b140640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa04406cb10 0x7fa0440cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa04b140640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0440d2df0 con 0x7fa04415e920 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa04b140640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa0440cf7a0 con 0x7fa0440d2120 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa04b140640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa0440cf190 con 0x7fa04406cb10 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0440d2120 0x7fa04406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0440d2120 0x7fa04406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43978/0 (socket says 172.21.15.156:43978) 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 -- 172.21.15.156:0/3533788777 learned_addr learned my addr 172.21.15.156:0/3533788777 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa0496dd640 1 --2- 172.21.15.156:0/3533788777 >> v2:172.21.15.203:3300/0 conn(0x7fa04406cb10 0x7fa0440cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 -- 172.21.15.156:0/3533788777 >> v2:172.21.15.203:3300/0 conn(0x7fa04406cb10 msgr2=0x7fa0440cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 --2- 172.21.15.156:0/3533788777 >> v2:172.21.15.203:3300/0 conn(0x7fa04406cb10 0x7fa0440cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 -- 172.21.15.156:0/3533788777 >> v1:172.21.15.203:6789/0 conn(0x7fa04415e920 legacy=0x7fa0440d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.203+0000 7fa043fff640 1 -- 172.21.15.156:0/3533788777 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0440d3080 con 0x7fa0440d2120 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.204+0000 7fa043fff640 1 --2- 172.21.15.156:0/3533788777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0440d2120 0x7fa04406c5d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa04408ca00 tx=0x7fa03802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de1ec4d1f3b0bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.204+0000 7fa0437fe640 1 -- 172.21.15.156:0/3533788777 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038035020 con 0x7fa0440d2120 2024-03-29T04:09:41.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.204+0000 7fa0437fe640 1 -- 172.21.15.156:0/3533788777 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa038031870 con 0x7fa0440d2120 2024-03-29T04:09:41.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.204+0000 7fa0437fe640 1 -- 172.21.15.156:0/3533788777 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038031c40 con 0x7fa0440d2120 2024-03-29T04:09:41.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 -- 172.21.15.156:0/3533788777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0440d2120 msgr2=0x7fa04406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 --2- 172.21.15.156:0/3533788777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0440d2120 0x7fa04406c5d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa04408ca00 tx=0x7fa03802b910 comp rx=0 tx=0).stop 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 -- 172.21.15.156:0/3533788777 shutdown_connections 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 --2- 172.21.15.156:0/3533788777 >> v2:172.21.15.203:3300/0 conn(0x7fa04406cb10 0x7fa0440cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 --2- 172.21.15.156:0/3533788777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0440d2120 0x7fa04406c5d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 -- 172.21.15.156:0/3533788777 >> 172.21.15.156:0/3533788777 conn(0x7fa04409d7a0 msgr2=0x7fa0440d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 -- 172.21.15.156:0/3533788777 shutdown_connections 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.205+0000 7fa04b140640 1 -- 172.21.15.156:0/3533788777 wait complete. 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.206+0000 7fa04b140640 1 Processor -- start 2024-03-29T04:09:41.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.206+0000 7fa04b140640 1 -- start start 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa04b140640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa04406cb10 0x7fa0440b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa04b140640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0440d2120 0x7fa0440b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa04b140640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 0x7fa0440b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa04b140640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa0440d34e0 con 0x7fa04415e920 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa04b140640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0440d3360 con 0x7fa04406cb10 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa04b140640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa0440d0170 con 0x7fa0440d2120 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 0x7fa0440b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 0x7fa0440b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43990/0 (socket says 172.21.15.156:43990) 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 -- 172.21.15.156:0/1939321155 learned_addr learned my addr 172.21.15.156:0/1939321155 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa048edc640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa04406cb10 0x7fa0440b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa043fff640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0440d2120 0x7fa0440b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 -- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0440d2120 msgr2=0x7fa0440b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0440d2120 0x7fa0440b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 -- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa04406cb10 msgr2=0x7fa0440b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa04406cb10 0x7fa0440b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.207+0000 7fa0496dd640 1 -- 172.21.15.156:0/1939321155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04415a6b0 con 0x7fa04415e920 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.208+0000 7fa0496dd640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 0x7fa0440b69c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa03400d830 tx=0x7fa03400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.208+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03400b6c0 con 0x7fa04415e920 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.208+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa034019070 con 0x7fa04415e920 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.208+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0440b93c0 con 0x7fa04415e920 2024-03-29T04:09:41.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.208+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03400bac0 con 0x7fa04415e920 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.208+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0440b3f10 con 0x7fa04415e920 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.209+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04415f840 con 0x7fa04415e920 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.209+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa03401e460 con 0x7fa04415e920 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.210+0000 7fa0417fa640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa03006c530 0x7fa03006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.210+0000 7fa048edc640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa03006c530 0x7fa03006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.210+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa03408bc20 con 0x7fa04415e920 2024-03-29T04:09:41.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.210+0000 7fa048edc640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa03006c530 0x7fa03006e9f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b450 tx=0x7fa03c00d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.218+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa04415f840 con 0x7fa04415e920 2024-03-29T04:09:41.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.399+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa04404f9d0 con 0x7fa04415e920 2024-03-29T04:09:41.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.399+0000 7fa0417fa640 1 -- 172.21.15.156:0/1939321155 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fa04415a6b0 con 0x7fa04415e920 2024-03-29T04:09:41.379 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:09:41.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.401+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa03006c530 msgr2=0x7fa03006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.401+0000 7fa04b140640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa03006c530 0x7fa03006e9f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b450 tx=0x7fa03c00d040 comp rx=0 tx=0).stop 2024-03-29T04:09:41.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.401+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 msgr2=0x7fa0440b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.401+0000 7fa04b140640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 0x7fa0440b69c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa03400d830 tx=0x7fa03400dcf0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.401+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 shutdown_connections 2024-03-29T04:09:41.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.401+0000 7fa04b140640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa03006c530 0x7fa03006e9f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.402+0000 7fa04b140640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04415e920 0x7fa0440b69c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.402+0000 7fa04b140640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0440d2120 0x7fa0440b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.402+0000 7fa04b140640 1 --2- 172.21.15.156:0/1939321155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa04406cb10 0x7fa0440b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.402+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 >> 172.21.15.156:0/1939321155 conn(0x7fa04409d7a0 msgr2=0x7fa04415ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:41.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.402+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 shutdown_connections 2024-03-29T04:09:41.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.402+0000 7fa04b140640 1 -- 172.21.15.156:0/1939321155 wait complete. 2024-03-29T04:09:41.396 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:09:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.524+0000 7f5aaca7e640 1 Processor -- start 2024-03-29T04:09:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.524+0000 7f5aaca7e640 1 -- start start 2024-03-29T04:09:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.525+0000 7f5aaca7e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 0x7f5aa80d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.525+0000 7f5aaca7e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5aa80d2120 0x7f5aa806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.525+0000 7f5aaca7e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa80cf190 con 0x7f5aa806cb10 2024-03-29T04:09:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.525+0000 7f5aaca7e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa80cf7a0 con 0x7f5aa80d2120 2024-03-29T04:09:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.525+0000 7f5aaca7e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa80d3080 con 0x7f5aa815e920 2024-03-29T04:09:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa659c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 0x7f5aa80d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa5d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5aa80d2120 0x7f5aa806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa6d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5aa806cb10 0x7f5aa80cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:42160/0 (socket says 172.21.15.156:42160) 2024-03-29T04:09:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa659c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 0x7f5aa80d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:52826/0 (socket says 172.21.15.156:52826) 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa5d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5aa80d2120 0x7f5aa806c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:52816/0 (socket says 172.21.15.156:52816) 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa6d9d640 1 -- 172.21.15.156:0/1327193709 learned_addr learned my addr 172.21.15.156:0/1327193709 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa5d9b640 1 -- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 msgr2=0x7f5aa80d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa5d9b640 1 --2- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 0x7f5aa80d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa5d9b640 1 -- 172.21.15.156:0/1327193709 >> v1:172.21.15.203:6789/0 conn(0x7f5aa806cb10 legacy=0x7f5aa80cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa5d9b640 1 -- 172.21.15.156:0/1327193709 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa80cf430 con 0x7f5aa80d2120 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa559a640 1 -- 172.21.15.156:0/1327193709 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833139126 0 0) 0x7f5aa80cf190 con 0x7f5aa806cb10 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.526+0000 7f5aa659c640 1 --2- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 0x7f5aa80d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.527+0000 7f5aa5d9b640 1 --2- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa80d2120 0x7f5aa806c5d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0097a0 tx=0x7f5a9c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bcc9932bb8ac681 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.527+0000 7f5aa559a640 1 -- 172.21.15.156:0/1327193709 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c035020 con 0x7f5aa80d2120 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.527+0000 7f5aa559a640 1 -- 172.21.15.156:0/1327193709 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9c031830 con 0x7f5aa80d2120 2024-03-29T04:09:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.527+0000 7f5aa559a640 1 -- 172.21.15.156:0/1327193709 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c031c00 con 0x7f5aa80d2120 2024-03-29T04:09:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.527+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa80d2120 msgr2=0x7f5aa806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa80d2120 0x7f5aa806c5d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0097a0 tx=0x7f5a9c02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1327193709 shutdown_connections 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa80d2120 0x7f5aa806c5d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1327193709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa815e920 0x7f5aa80d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1327193709 >> 172.21.15.156:0/1327193709 conn(0x7f5aa809d7a0 msgr2=0x7f5aa80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1327193709 shutdown_connections 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.528+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1327193709 wait complete. 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.529+0000 7f5aaca7e640 1 Processor -- start 2024-03-29T04:09:41.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.529+0000 7f5aaca7e640 1 -- start start 2024-03-29T04:09:41.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.530+0000 7f5aaca7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa806cb10 0x7f5aa81e8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.530+0000 7f5aaca7e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa80d2120 0x7f5aa81612b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.530+0000 7f5aaca7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 0x7f5aa81617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.530+0000 7f5aaca7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa80d04c0 con 0x7f5aa815e920 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.530+0000 7f5aaca7e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa80d0340 con 0x7f5aa806cb10 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.530+0000 7f5aaca7e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa80d2c20 con 0x7f5aa80d2120 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 0x7f5aa81617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa806cb10 0x7f5aa81e8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 0x7f5aa81617f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39604/0 (socket says 172.21.15.156:39604) 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa806cb10 0x7f5aa81e8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:52820/0 (socket says 172.21.15.156:52820) 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 -- 172.21.15.156:0/1653747827 learned_addr learned my addr 172.21.15.156:0/1653747827 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa5d9b640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa80d2120 0x7f5aa81612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 -- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa80d2120 msgr2=0x7f5aa81612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa80d2120 0x7f5aa81612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 -- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa806cb10 msgr2=0x7f5aa81e8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa806cb10 0x7f5aa81e8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 -- 172.21.15.156:0/1653747827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa81eb0f0 con 0x7f5aa815e920 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aa6d9d640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 0x7f5aa81617f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c4b0 tx=0x7f5a9800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98018070 con 0x7f5aa815e920 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a980074e0 con 0x7f5aa815e920 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98004a20 con 0x7f5aa815e920 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa81ea570 con 0x7f5aa815e920 2024-03-29T04:09:41.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.531+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa81eab30 con 0x7f5aa815e920 2024-03-29T04:09:41.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.533+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa8068fc0 con 0x7f5aa815e920 2024-03-29T04:09:41.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.533+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5a9801b070 con 0x7f5aa815e920 2024-03-29T04:09:41.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.534+0000 7f5a96ffd640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8806c530 0x7f5a8806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.534+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5a9808a960 con 0x7f5aa815e920 2024-03-29T04:09:41.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.534+0000 7f5aa659c640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8806c530 0x7f5a8806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.534+0000 7f5aa659c640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8806c530 0x7f5a8806e9f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009bd0 tx=0x7f5aa000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.542+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5aa8068fc0 con 0x7f5aa815e920 2024-03-29T04:09:41.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.720+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5aa80a2480 con 0x7f5aa815e920 2024-03-29T04:09:41.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.721+0000 7f5a96ffd640 1 -- 172.21.15.156:0/1653747827 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f5a980588b0 con 0x7f5aa815e920 2024-03-29T04:09:41.701 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:09:41.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8806c530 msgr2=0x7f5a8806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8806c530 0x7f5a8806e9f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009bd0 tx=0x7f5aa000b040 comp rx=0 tx=0).stop 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 msgr2=0x7f5aa81617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 0x7f5aa81617f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c4b0 tx=0x7f5a9800c970 comp rx=0 tx=0).stop 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 shutdown_connections 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8806c530 0x7f5a8806e9f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa815e920 0x7f5aa81617f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa80d2120 0x7f5aa81612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 --2- 172.21.15.156:0/1653747827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa806cb10 0x7f5aa81e8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 >> 172.21.15.156:0/1653747827 conn(0x7f5aa809d7a0 msgr2=0x7f5aa80b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 shutdown_connections 2024-03-29T04:09:41.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.723+0000 7f5aaca7e640 1 -- 172.21.15.156:0/1653747827 wait complete. 2024-03-29T04:09:41.713 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:41.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.838+0000 7f74bf300640 1 Processor -- start 2024-03-29T04:09:41.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.838+0000 7f74bf300640 1 -- start start 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bf300640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f74b806f7a0 0x7f74b81661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bf300640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74b8072760 con 0x7f74b80af6b0 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bf300640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74b80afa90 con 0x7f74b80b3050 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bf300640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f74b80b3510 con 0x7f74b806f7a0 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bd09c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f74b80af6b0 0x7f74b815c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44692/0 (socket says 172.21.15.156:44692) 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bd09c640 1 -- 172.21.15.156:0/2793520387 learned_addr learned my addr 172.21.15.156:0/2793520387 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74bd89d640 1 --2- 172.21.15.156:0/2793520387 >> v2:172.21.15.156:3300/0 conn(0x7f74b806f7a0 0x7f74b81661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74affff640 1 -- 172.21.15.156:0/2793520387 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963139234 0 0) 0x7f74b4001110 con 0x7f74b80af6b0 2024-03-29T04:09:41.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.839+0000 7f74affff640 1 -- 172.21.15.156:0/2793520387 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74a4003520 con 0x7f74b80af6b0 2024-03-29T04:09:41.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.840+0000 7f74bd89d640 1 -- 172.21.15.156:0/2793520387 >> v1:172.21.15.203:6789/0 conn(0x7f74b80b3050 legacy=0x7f74b806f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.840+0000 7f74bd89d640 1 -- 172.21.15.156:0/2793520387 >> v1:172.21.15.156:6789/0 conn(0x7f74b80af6b0 legacy=0x7f74b815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.840+0000 7f74bd89d640 1 -- 172.21.15.156:0/2793520387 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b80b37f0 con 0x7f74b806f7a0 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74bd89d640 1 --2- 172.21.15.156:0/2793520387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b806f7a0 0x7f74b81661a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009d90 tx=0x7f74b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d0210f9915dca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74affff640 1 -- 172.21.15.156:0/2793520387 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0039070 con 0x7f74b806f7a0 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74affff640 1 -- 172.21.15.156:0/2793520387 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b002cc60 con 0x7f74b806f7a0 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74affff640 1 -- 172.21.15.156:0/2793520387 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0034660 con 0x7f74b806f7a0 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74bf300640 1 -- 172.21.15.156:0/2793520387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b806f7a0 msgr2=0x7f74b81661a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74bf300640 1 --2- 172.21.15.156:0/2793520387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b806f7a0 0x7f74b81661a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009d90 tx=0x7f74b002beb0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74bf300640 1 -- 172.21.15.156:0/2793520387 shutdown_connections 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74bf300640 1 --2- 172.21.15.156:0/2793520387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b806f7a0 0x7f74b81661a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.841+0000 7f74bf300640 1 -- 172.21.15.156:0/2793520387 >> 172.21.15.156:0/2793520387 conn(0x7f74b80b1d50 msgr2=0x7f74b80b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:41.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.842+0000 7f74bf300640 1 -- 172.21.15.156:0/2793520387 shutdown_connections 2024-03-29T04:09:41.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.842+0000 7f74bf300640 1 -- 172.21.15.156:0/2793520387 wait complete. 2024-03-29T04:09:41.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 Processor -- start 2024-03-29T04:09:41.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 -- start start 2024-03-29T04:09:41.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74b806f7a0 0x7f74b81510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74b80af6b0 0x7f74b81e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 0x7f74b8154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f74b80b4090 con 0x7f74b80b3050 2024-03-29T04:09:41.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74b80b3db0 con 0x7f74b806f7a0 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bf300640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f74b80b45b0 con 0x7f74b80af6b0 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.843+0000 7f74bd89d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 0x7f74b8154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 0x7f74b8154330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39634/0 (socket says 172.21.15.156:39634) 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 -- 172.21.15.156:0/3888476440 learned_addr learned my addr 172.21.15.156:0/3888476440 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd09c640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74b806f7a0 0x7f74b81510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bc89b640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74b80af6b0 0x7f74b81e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 -- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74b80af6b0 msgr2=0x7f74b81e3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74b80af6b0 0x7f74b81e3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 -- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74b806f7a0 msgr2=0x7f74b81510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74b806f7a0 0x7f74b81510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 -- 172.21.15.156:0/3888476440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b81e2c80 con 0x7f74b80b3050 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bc89b640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74b80af6b0 0x7f74b81e3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd09c640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74b806f7a0 0x7f74b81510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bd89d640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 0x7f74b8154330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009d90 tx=0x7f74b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0039050 con 0x7f74b80b3050 2024-03-29T04:09:41.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b0045070 con 0x7f74b80b3050 2024-03-29T04:09:41.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0009450 con 0x7f74b80b3050 2024-03-29T04:09:41.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b81515e0 con 0x7f74b80b3050 2024-03-29T04:09:41.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.844+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b81e3200 con 0x7f74b80b3050 2024-03-29T04:09:41.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.846+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b8068610 con 0x7f74b80b3050 2024-03-29T04:09:41.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.846+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f74b003e850 con 0x7f74b80b3050 2024-03-29T04:09:41.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.846+0000 7f74adffb640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f749c06c530 0x7f749c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:41.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.846+0000 7f74bd09c640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f749c06c530 0x7f749c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:41.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.846+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f74b00ae280 con 0x7f74b80b3050 2024-03-29T04:09:41.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.847+0000 7f74bd09c640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f749c06c530 0x7f749c06e9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74b4007fd0 tx=0x7f74b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:41.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:41.851+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f74b0032e20 con 0x7f74b80b3050 2024-03-29T04:09:42.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.115+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74b80aeb40 con 0x7f74b80b3050 2024-03-29T04:09:42.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.116+0000 7f74adffb640 1 -- 172.21.15.156:0/3888476440 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f74b007c940 con 0x7f74b80b3050 2024-03-29T04:09:42.096 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:42.096 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:42.096 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:42.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f749c06c530 msgr2=0x7f749c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f749c06c530 0x7f749c06e9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74b4007fd0 tx=0x7f74b4009040 comp rx=0 tx=0).stop 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 msgr2=0x7f74b8154330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 0x7f74b8154330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009d90 tx=0x7f74b0004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 shutdown_connections 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f749c06c530 0x7f749c06e9f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74b80b3050 0x7f74b8154330 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74b80af6b0 0x7f74b81e3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 --2- 172.21.15.156:0/3888476440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74b806f7a0 0x7f74b81510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 >> 172.21.15.156:0/3888476440 conn(0x7f74b80b1d50 msgr2=0x7f74b80b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 shutdown_connections 2024-03-29T04:09:42.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.119+0000 7f74bf300640 1 -- 172.21.15.156:0/3888476440 wait complete. 2024-03-29T04:09:42.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:42.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.232+0000 7fa67f6f8640 1 Processor -- start 2024-03-29T04:09:42.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.232+0000 7fa67f6f8640 1 -- start start 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67f6f8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 0x7fa6780cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67f6f8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa6780cc870 0x7fa6780cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67f6f8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67815cb50 con 0x7fa6781609b0 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67f6f8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa6780cf8b0 con 0x7fa6780cc870 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67f6f8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa6780cd200 con 0x7fa6781568f0 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67cc93640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa6781609b0 0x7fa67815d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44700/0 (socket says 172.21.15.156:44700) 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67cc93640 1 -- 172.21.15.156:0/1979912127 learned_addr learned my addr 172.21.15.156:0/1979912127 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105160201 0 0) 0x7fa67815cb50 con 0x7fa6781609b0 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa664003520 con 0x7fa6781609b0 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67dc95640 1 --2- 172.21.15.156:0/1979912127 >> v2:172.21.15.203:3300/0 conn(0x7fa6780cc870 0x7fa6780cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa67d494640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 0x7fa6780cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4241614316 0 0) 0x7fa664003520 con 0x7fa6781609b0 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 msgr2=0x7fa6780cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 0x7fa6780cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 >> v2:172.21.15.203:3300/0 conn(0x7fa6780cc870 msgr2=0x7fa6780cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 --2- 172.21.15.156:0/1979912127 >> v2:172.21.15.203:3300/0 conn(0x7fa6780cc870 0x7fa6780cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa678160e30 con 0x7fa6781609b0 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fa668001690 con 0x7fa6781609b0 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 >> v1:172.21.15.156:6789/0 conn(0x7fa6781609b0 legacy=0x7fa67815d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640039b0 0x7fa664003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 0x7fa664004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa664004d40 0x7fa6640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cf8b0 con 0x7fa664004d40 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cd200 con 0x7fa6640039b0 2024-03-29T04:09:42.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.233+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa66400bb20 con 0x7fa6781568f0 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67dc95640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa664004d40 0x7fa6640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67d494640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640039b0 0x7fa664003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67cc93640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 0x7fa664004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67dc95640 1 -- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 msgr2=0x7fa664004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67dc95640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6781568f0 0x7fa664004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67dc95640 1 -- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640039b0 msgr2=0x7fa664003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67dc95640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640039b0 0x7fa664003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.234+0000 7fa67dc95640 1 -- 172.21.15.156:0/1979912127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa664003520 con 0x7fa664004d40 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67dc95640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa664004d40 0x7fa6640051c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa67000ba30 tx=0x7fa664005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9913d5796c30c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670036020 con 0x7fa664004d40 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa670032680 con 0x7fa664004d40 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa66ffff640 1 -- 172.21.15.156:0/1979912127 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67000b600 con 0x7fa664004d40 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67f6f8640 1 -- 172.21.15.156:0/1979912127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa664004d40 msgr2=0x7fa6640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/1979912127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa664004d40 0x7fa6640051c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa67000ba30 tx=0x7fa664005830 comp rx=0 tx=0).stop 2024-03-29T04:09:42.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67dc95640 1 -- 172.21.15.156:0/1979912127 reap_dead start 2024-03-29T04:09:42.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67f6f8640 1 -- 172.21.15.156:0/1979912127 shutdown_connections 2024-03-29T04:09:42.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67f6f8640 1 -- 172.21.15.156:0/1979912127 >> 172.21.15.156:0/1979912127 conn(0x7fa6780b3eb0 msgr2=0x7fa6780b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:42.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.235+0000 7fa67f6f8640 1 -- 172.21.15.156:0/1979912127 shutdown_connections 2024-03-29T04:09:42.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.236+0000 7fa67f6f8640 1 -- 172.21.15.156:0/1979912127 wait complete. 2024-03-29T04:09:42.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 Processor -- start 2024-03-29T04:09:42.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 -- start start 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa67814f5c0 0x7fa6780d0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 0x7fa6780cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6780cb560 0x7fa6780c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa67815cdf0 con 0x7fa6780d0bb0 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cf5d0 con 0x7fa6780cb560 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67f6f8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa678156cd0 con 0x7fa67814f5c0 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.237+0000 7fa67cc93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 0x7fa6780cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 0x7fa6780cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39660/0 (socket says 172.21.15.156:39660) 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 -- 172.21.15.156:0/2327089140 learned_addr learned my addr 172.21.15.156:0/2327089140 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67dc95640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6780cb560 0x7fa6780c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67d494640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa67814f5c0 0x7fa6780d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 -- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa67814f5c0 msgr2=0x7fa6780d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa67814f5c0 0x7fa6780d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 -- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6780cb560 msgr2=0x7fa6780c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6780cb560 0x7fa6780c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 -- 172.21.15.156:0/2327089140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67814f9a0 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67cc93640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 0x7fa6780cb020 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa6680072f0 tx=0x7fa6680077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66801a070 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66800f960 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6780cbc00 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66800fd30 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.238+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6781f2860 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.239+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa678068610 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.240+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa66801e3f0 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.240+0000 7fa66dffb640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa65c06c560 0x7fa65c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.240+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa66808b050 con 0x7fa6780d0bb0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.240+0000 7fa67d494640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa65c06c560 0x7fa65c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.240+0000 7fa67d494640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa65c06c560 0x7fa65c06ea20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6780d3090 tx=0x7fa674009760 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.247+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa6680595f0 con 0x7fa6780d0bb0 2024-03-29T04:09:42.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.428+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6780cb940 con 0x7fa6780d0bb0 2024-03-29T04:09:42.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.429+0000 7fa66dffb640 1 -- 172.21.15.156:0/2327089140 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16878 (secure 0 0 0) 0x7fa668058ef0 con 0x7fa6780d0bb0 2024-03-29T04:09:42.409 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:42.409 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":23,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:34.930341+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:42.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa65c06c560 msgr2=0x7fa65c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa65c06c560 0x7fa65c06ea20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6780d3090 tx=0x7fa674009760 comp rx=0 tx=0).stop 2024-03-29T04:09:42.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 msgr2=0x7fa6780cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 0x7fa6780cb020 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa6680072f0 tx=0x7fa6680077b0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 shutdown_connections 2024-03-29T04:09:42.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa65c06c560 0x7fa65c06ea20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6780cb560 0x7fa6780c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6780d0bb0 0x7fa6780cb020 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.431+0000 7fa67f6f8640 1 --2- 172.21.15.156:0/2327089140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa67814f5c0 0x7fa6780d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.432+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 >> 172.21.15.156:0/2327089140 conn(0x7fa6780b3eb0 msgr2=0x7fa6780cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:42.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.432+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 shutdown_connections 2024-03-29T04:09:42.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.432+0000 7fa67f6f8640 1 -- 172.21.15.156:0/2327089140 wait complete. 2024-03-29T04:09:42.422 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T04:09:42.422 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:42.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.545+0000 7f94f404a640 1 Processor -- start 2024-03-29T04:09:42.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.545+0000 7f94f404a640 1 -- start start 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f404a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94ec15e730 0x7f94ec0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f404a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec06cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f404a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94ec0cfa20 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f404a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f94ec06c720 con 0x7f94ec15e730 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f404a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f94ec0cfd00 con 0x7f94ec0d49d0 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f1de6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f94ec15e730 0x7f94ec0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f15e5640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f94ec0cf3e0 0x7f94ec0d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44710/0 (socket says 172.21.15.156:44710) 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f15e5640 1 -- 172.21.15.156:0/2737782829 learned_addr learned my addr 172.21.15.156:0/2737782829 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f25e7640 1 --2- 172.21.15.156:0/2737782829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec06cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f1de6640 1 -- 172.21.15.156:0/2737782829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 msgr2=0x7f94ec06cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:42.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f1de6640 1 --2- 172.21.15.156:0/2737782829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec06cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f1de6640 1 -- 172.21.15.156:0/2737782829 >> v1:172.21.15.156:6789/0 conn(0x7f94ec0cf3e0 legacy=0x7f94ec0d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f1de6640 1 -- 172.21.15.156:0/2737782829 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ec0cffe0 con 0x7f94ec15e730 2024-03-29T04:09:42.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.546+0000 7f94f0de4640 1 -- 172.21.15.156:0/2737782829 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422493012 0 0) 0x7f94ec0cfa20 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.547+0000 7f94f1de6640 1 --2- 172.21.15.156:0/2737782829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec15e730 0x7f94ec0ceea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f94e80097a0 tx=0x7f94e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4150dfa543fffd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.547+0000 7f94f0de4640 1 -- 172.21.15.156:0/2737782829 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8035020 con 0x7f94ec15e730 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.547+0000 7f94f0de4640 1 -- 172.21.15.156:0/2737782829 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94e8031830 con 0x7f94ec15e730 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.547+0000 7f94f0de4640 1 -- 172.21.15.156:0/2737782829 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8031c00 con 0x7f94ec15e730 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 -- 172.21.15.156:0/2737782829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec15e730 msgr2=0x7f94ec0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 --2- 172.21.15.156:0/2737782829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec15e730 0x7f94ec0ceea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f94e80097a0 tx=0x7f94e802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 -- 172.21.15.156:0/2737782829 shutdown_connections 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 --2- 172.21.15.156:0/2737782829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec06cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 --2- 172.21.15.156:0/2737782829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec15e730 0x7f94ec0ceea0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 -- 172.21.15.156:0/2737782829 >> 172.21.15.156:0/2737782829 conn(0x7f94ec08bfd0 msgr2=0x7f94ec09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 -- 172.21.15.156:0/2737782829 shutdown_connections 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.548+0000 7f94f404a640 1 -- 172.21.15.156:0/2737782829 wait complete. 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.549+0000 7f94f404a640 1 Processor -- start 2024-03-29T04:09:42.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.549+0000 7f94f404a640 1 -- start start 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 0x7f94ec0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94ec15e730 0x7f94ec0ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94ec15eb10 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94ec0d2220 con 0x7f94ec15e730 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f94ec15ec90 con 0x7f94ec0d49d0 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 0x7f94ec0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 0x7f94ec0ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39664/0 (socket says 172.21.15.156:39664) 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 -- 172.21.15.156:0/3305644542 learned_addr learned my addr 172.21.15.156:0/3305644542 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 -- 172.21.15.156:0/3305644542 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 msgr2=0x7f94ec0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 -- 172.21.15.156:0/3305644542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94ec15e730 msgr2=0x7f94ec0ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94ec15e730 0x7f94ec0ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 -- 172.21.15.156:0/3305644542 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ec0c93f0 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f1de6640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 0x7f94ec0ca490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f94e80098c0 tx=0x7f94e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8040070 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94e803e070 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94ec0c96d0 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8035040 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.550+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94ec0c9c90 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.552+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94ec068610 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.552+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f94e8038020 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.552+0000 7f94e27fc640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94d006c530 0x7f94d006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.552+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f94e80acc30 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.552+0000 7f94f15e5640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94d006c530 0x7f94d006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.553+0000 7f94f15e5640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94d006c530 0x7f94d006e9f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94ec15f6a0 tx=0x7f94dc0083a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.561+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f94e8031800 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.789+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94ec0caf60 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.789+0000 7f94e27fc640 1 -- 172.21.15.156:0/3305644542 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f94e807b1d0 con 0x7f94ec0cf3e0 2024-03-29T04:09:42.769 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:42.769 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:42.769 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:42.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94d006c530 msgr2=0x7f94d006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94d006c530 0x7f94d006e9f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94ec15f6a0 tx=0x7f94dc0083a0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.771 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 msgr2=0x7f94ec0ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 0x7f94ec0ca490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f94e80098c0 tx=0x7f94e80046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 shutdown_connections 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f94d006c530 0x7f94d006e9f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94ec15e730 0x7f94ec0ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94ec0d49d0 0x7f94ec0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 --2- 172.21.15.156:0/3305644542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94ec0cf3e0 0x7f94ec0ca490 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 >> 172.21.15.156:0/3305644542 conn(0x7f94ec08bfd0 msgr2=0x7f94ec06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 shutdown_connections 2024-03-29T04:09:42.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.792+0000 7f94f404a640 1 -- 172.21.15.156:0/3305644542 wait complete. 2024-03-29T04:09:42.782 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:42.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.903+0000 7f8f67d1b640 1 Processor -- start 2024-03-29T04:09:42.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.903+0000 7f8f67d1b640 1 -- start start 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f67d1b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8f6015e730 0x7f8f600ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f67d1b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f600cfa20 con 0x7f8f600cf3e0 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f67d1b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f600cfd00 con 0x7f8f600d49d0 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f67d1b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8f6006c720 con 0x7f8f6015e730 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f65ab7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8f6015e730 0x7f8f600ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f652b6640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8f600cf3e0 0x7f8f600d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44714/0 (socket says 172.21.15.156:44714) 2024-03-29T04:09:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.904+0000 7f8f652b6640 1 -- 172.21.15.156:0/3337212666 learned_addr learned my addr 172.21.15.156:0/3337212666 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.905+0000 7f8f65ab7640 1 -- 172.21.15.156:0/3337212666 >> v1:172.21.15.203:6789/0 conn(0x7f8f600d49d0 legacy=0x7f8f6006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.905+0000 7f8f65ab7640 1 -- 172.21.15.156:0/3337212666 >> v1:172.21.15.156:6789/0 conn(0x7f8f600cf3e0 legacy=0x7f8f600d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.905+0000 7f8f64ab5640 1 -- 172.21.15.156:0/3337212666 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4210614206 0 0) 0x7f8f600cfa20 con 0x7f8f600cf3e0 2024-03-29T04:09:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.905+0000 7f8f65ab7640 1 -- 172.21.15.156:0/3337212666 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f600cffe0 con 0x7f8f6015e730 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.905+0000 7f8f65ab7640 1 --2- 172.21.15.156:0/3337212666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f600ceea0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8f580097a0 tx=0x7f8f5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef03d3c18c7b01bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f64ab5640 1 -- 172.21.15.156:0/3337212666 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58035020 con 0x7f8f6015e730 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f64ab5640 1 -- 172.21.15.156:0/3337212666 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f58031830 con 0x7f8f6015e730 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f64ab5640 1 -- 172.21.15.156:0/3337212666 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58031c00 con 0x7f8f6015e730 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 -- 172.21.15.156:0/3337212666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 msgr2=0x7f8f600ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/3337212666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f600ceea0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8f580097a0 tx=0x7f8f5802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 -- 172.21.15.156:0/3337212666 shutdown_connections 2024-03-29T04:09:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/3337212666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f600ceea0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 -- 172.21.15.156:0/3337212666 >> 172.21.15.156:0/3337212666 conn(0x7f8f6008bfd0 msgr2=0x7f8f6009d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 -- 172.21.15.156:0/3337212666 shutdown_connections 2024-03-29T04:09:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.906+0000 7f8f67d1b640 1 -- 172.21.15.156:0/3337212666 wait complete. 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.907+0000 7f8f67d1b640 1 Processor -- start 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.907+0000 7f8f67d1b640 1 -- start start 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f67d1b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f600cf3e0 0x7f8f60156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f67d1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f600d49d0 0x7f8f600ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f67d1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f601604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f67d1b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6015eb10 con 0x7f8f6015e730 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f67d1b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f600d2220 con 0x7f8f600d49d0 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f67d1b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6015ec90 con 0x7f8f600cf3e0 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f601604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f601604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39682/0 (socket says 172.21.15.156:39682) 2024-03-29T04:09:42.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 -- 172.21.15.156:0/842234456 learned_addr learned my addr 172.21.15.156:0/842234456 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f652b6640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f600d49d0 0x7f8f600ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f65ab7640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f600cf3e0 0x7f8f60156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 -- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f600cf3e0 msgr2=0x7f8f60156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f600cf3e0 0x7f8f60156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 -- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f600d49d0 msgr2=0x7f8f600ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f600d49d0 0x7f8f600ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.908+0000 7f8f662b8640 1 -- 172.21.15.156:0/842234456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f601f7780 con 0x7f8f6015e730 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.909+0000 7f8f662b8640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f601604c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c7a0 tx=0x7f8f5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.909+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c003b70 con 0x7f8f6015e730 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.909+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c003d30 con 0x7f8f6015e730 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.909+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f601f7a60 con 0x7f8f6015e730 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.909+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00aa90 con 0x7f8f6015e730 2024-03-29T04:09:42.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.909+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f601f8020 con 0x7f8f6015e730 2024-03-29T04:09:42.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.910+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8f5c00ac30 con 0x7f8f6015e730 2024-03-29T04:09:42.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.910+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f60068610 con 0x7f8f6015e730 2024-03-29T04:09:42.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.911+0000 7f8f567fc640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8f4406c530 0x7f8f4406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:42.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.911+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8f5c08b660 con 0x7f8f6015e730 2024-03-29T04:09:42.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.911+0000 7f8f65ab7640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8f4406c530 0x7f8f4406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:42.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.911+0000 7f8f65ab7640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8f4406c530 0x7f8f4406e9f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f58036000 tx=0x7f8f5803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:42.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:42.919+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8f5c059c00 con 0x7f8f6015e730 2024-03-29T04:09:43.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.140+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f600b3ec0 con 0x7f8f6015e730 2024-03-29T04:09:43.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.140+0000 7f8f567fc640 1 -- 172.21.15.156:0/842234456 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5184 (secure 0 0 0) 0x7f8f5c059500 con 0x7f8f6015e730 2024-03-29T04:09:43.120 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:43.120 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/802079400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":802079400},{"type":"v1","addr":"172.21.15.203:6835","nonce":802079400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2405943227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2405943227},{"type":"v1","addr":"172.21.15.203:6833","nonce":2405943227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2220570215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2220570215},{"type":"v1","addr":"172.21.15.156:6835","nonce":2220570215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:08:53.151747+0000","modified":"2024-03-29T04:08:56.192682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/2168836037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2168836037},{"type":"v1","addr":"172.21.15.156:6837","nonce":2168836037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-29T04:09:43.120 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8f4406c530 msgr2=0x7f8f4406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8f4406c530 0x7f8f4406e9f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f58036000 tx=0x7f8f5803c040 comp rx=0 tx=0).stop 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 msgr2=0x7f8f601604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f601604c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c7a0 tx=0x7f8f5c00cc60 comp rx=0 tx=0).stop 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 shutdown_connections 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8f4406c530 0x7f8f4406e9f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6015e730 0x7f8f601604c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f600d49d0 0x7f8f600ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 --2- 172.21.15.156:0/842234456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f600cf3e0 0x7f8f60156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 >> 172.21.15.156:0/842234456 conn(0x7f8f6008bfd0 msgr2=0x7f8f600b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:43.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 shutdown_connections 2024-03-29T04:09:43.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.142+0000 7f8f67d1b640 1 -- 172.21.15.156:0/842234456 wait complete. 2024-03-29T04:09:43.133 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:43.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.265+0000 7f73a07a8640 1 Processor -- start 2024-03-29T04:09:43.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.265+0000 7f73a07a8640 1 -- start start 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f73a07a8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f739806b5b0 0x7f739806ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f73a07a8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 0x7f73980ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f73a07a8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73980b65c0 con 0x7f739815e750 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f73a07a8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f739806fa90 con 0x7f739806b5b0 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f73a07a8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f739806fce0 con 0x7f73980ae390 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f739806b5b0 0x7f739806ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f739806b5b0 0x7f739806ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39690/0 (socket says 172.21.15.156:39690) 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 -- 172.21.15.156:0/163982773 learned_addr learned my addr 172.21.15.156:0/163982773 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 -- 172.21.15.156:0/163982773 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 msgr2=0x7f73980ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 --2- 172.21.15.156:0/163982773 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 0x7f73980ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 -- 172.21.15.156:0/163982773 >> v1:172.21.15.203:6789/0 conn(0x7f739815e750 legacy=0x7f73980b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.266+0000 7f739dd43640 1 -- 172.21.15.156:0/163982773 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739806ffc0 con 0x7f739806b5b0 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.267+0000 7f739dd43640 1 --2- 172.21.15.156:0/163982773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f739806ba40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7398071170 tx=0x7f738802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc0f9722f6e2228 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.267+0000 7f739d542640 1 -- 172.21.15.156:0/163982773 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388035020 con 0x7f739806b5b0 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.267+0000 7f739d542640 1 -- 172.21.15.156:0/163982773 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7388031640 con 0x7f739806b5b0 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.267+0000 7f739d542640 1 -- 172.21.15.156:0/163982773 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738802d040 con 0x7f739806b5b0 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 -- 172.21.15.156:0/163982773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 msgr2=0x7f739806ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 --2- 172.21.15.156:0/163982773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f739806ba40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7398071170 tx=0x7f738802b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 -- 172.21.15.156:0/163982773 shutdown_connections 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 --2- 172.21.15.156:0/163982773 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 0x7f73980ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 --2- 172.21.15.156:0/163982773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f739806ba40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 -- 172.21.15.156:0/163982773 >> 172.21.15.156:0/163982773 conn(0x7f7398072d10 msgr2=0x7f73980b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:43.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 -- 172.21.15.156:0/163982773 shutdown_connections 2024-03-29T04:09:43.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.268+0000 7f73a07a8640 1 -- 172.21.15.156:0/163982773 wait complete. 2024-03-29T04:09:43.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.269+0000 7f73a07a8640 1 Processor -- start 2024-03-29T04:09:43.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.269+0000 7f73a07a8640 1 -- start start 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f7398161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 0x7f73981e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f739815e750 0x7f739814a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7398070860 con 0x7f739806b5b0 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7398070580 con 0x7f739815e750 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f739809ac60 con 0x7f73980ae390 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f7398161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f7398161140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39702/0 (socket says 172.21.15.156:39702) 2024-03-29T04:09:43.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 -- 172.21.15.156:0/3207198247 learned_addr learned my addr 172.21.15.156:0/3207198247 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 -- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 msgr2=0x7f73981e6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739ed45640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f739815e750 0x7f739814a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 0x7f73981e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 -- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f739815e750 msgr2=0x7f739814a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f739815e750 0x7f739814a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 -- 172.21.15.156:0/3207198247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73981e8100 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f739e544640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f7398161140 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f739400c440 tx=0x7f739400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394017070 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73940035f0 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73981e7860 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73940039e0 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.270+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73981e7e20 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.271+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7398068610 con 0x7f739806b5b0 2024-03-29T04:09:43.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.272+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f739401a070 con 0x7f739806b5b0 2024-03-29T04:09:43.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.272+0000 7f738effd640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f737c06c530 0x7f737c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.272+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f739408abe0 con 0x7f739806b5b0 2024-03-29T04:09:43.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.272+0000 7f739dd43640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f737c06c530 0x7f737c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.273+0000 7f739dd43640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f737c06c530 0x7f737c06e9f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7388036000 tx=0x7f7388030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.277+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7394059180 con 0x7f739806b5b0 2024-03-29T04:09:43.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.459+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73980b0ae0 con 0x7f739806b5b0 2024-03-29T04:09:43.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.460+0000 7f738effd640 1 -- 172.21.15.156:0/3207198247 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16878 (secure 0 0 0) 0x7f73940010c0 con 0x7f739806b5b0 2024-03-29T04:09:43.440 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:43.440 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":23,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:34.930341+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:08:51.621554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T04:08:52.104255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f737c06c530 msgr2=0x7f737c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f737c06c530 0x7f737c06e9f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7388036000 tx=0x7f7388030000 comp rx=0 tx=0).stop 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 msgr2=0x7f7398161140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f7398161140 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f739400c440 tx=0x7f739400c900 comp rx=0 tx=0).stop 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 shutdown_connections 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f737c06c530 0x7f737c06e9f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f739815e750 0x7f739814a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73980ae390 0x7f73981e6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 --2- 172.21.15.156:0/3207198247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739806b5b0 0x7f7398161140 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 >> 172.21.15.156:0/3207198247 conn(0x7f7398072d10 msgr2=0x7f739806e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:43.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 shutdown_connections 2024-03-29T04:09:43.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.462+0000 7f73a07a8640 1 -- 172.21.15.156:0/3207198247 wait complete. 2024-03-29T04:09:43.454 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T04:09:43.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.577+0000 7fc19a540640 1 Processor -- start 2024-03-29T04:09:43.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.577+0000 7fc19a540640 1 -- start start 2024-03-29T04:09:43.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc19a540640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc19415e920 0x7fc1940d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc19a540640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940d2120 0x7fc19406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc19a540640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1940cf190 con 0x7fc19406cb10 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc19a540640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc1940cf7a0 con 0x7fc19415e920 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc19a540640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc1940d3080 con 0x7fc1940d2120 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc198add640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc19406cb10 0x7fc1940cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44720/0 (socket says 172.21.15.156:44720) 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.578+0000 7fc198add640 1 -- 172.21.15.156:0/873223128 learned_addr learned my addr 172.21.15.156:0/873223128 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc1937fe640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940d2120 0x7fc19406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 --2- 172.21.15.156:0/873223128 >> v2:172.21.15.203:3300/0 conn(0x7fc19415e920 0x7fc1940d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582981039 0 0) 0x7fc1940cf190 con 0x7fc19406cb10 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc17c003520 con 0x7fc19406cb10 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 36867486 0 0) 0x7fc17c003520 con 0x7fc19406cb10 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940d2120 msgr2=0x7fc19406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940d2120 0x7fc19406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 >> v2:172.21.15.203:3300/0 conn(0x7fc19415e920 msgr2=0x7fc1940d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 --2- 172.21.15.156:0/873223128 >> v2:172.21.15.203:3300/0 conn(0x7fc19415e920 0x7fc1940d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1940cf430 con 0x7fc19406cb10 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fc18c0019f0 con 0x7fc19406cb10 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 >> v1:172.21.15.156:6789/0 conn(0x7fc19406cb10 legacy=0x7fc1940cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc17c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc17c004800 0x7fc17c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc17c0051c0 0x7fc17c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940cf7a0 con 0x7fc17c0039b0 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940d3080 con 0x7fc17c004800 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c00bf30 con 0x7fc17c0051c0 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc1937fe640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940d2120 0x7fc19406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc17c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc198add640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc17c0051c0 0x7fc17c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc1937fe640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc17c004800 0x7fc17c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 -- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc17c0051c0 msgr2=0x7fc17c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc17c0051c0 0x7fc17c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 -- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc17c004800 msgr2=0x7fc17c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc17c004800 0x7fc17c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.579+0000 7fc193fff640 1 -- 172.21.15.156:0/873223128 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c003520 con 0x7fc17c0039b0 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.580+0000 7fc193fff640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc17c003dd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ac0 tx=0x7fc17c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a34ddd0a0cdc37 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.580+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184039070 con 0x7fc17c0039b0 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.580+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18402c6d0 con 0x7fc17c0039b0 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.580+0000 7fc192ffd640 1 -- 172.21.15.156:0/873223128 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184009690 con 0x7fc17c0039b0 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc19a540640 1 -- 172.21.15.156:0/873223128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 msgr2=0x7fc17c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc19a540640 1 --2- 172.21.15.156:0/873223128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc17c003dd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ac0 tx=0x7fc17c005830 comp rx=0 tx=0).stop 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc198add640 1 -- 172.21.15.156:0/873223128 reap_dead start 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc19a540640 1 -- 172.21.15.156:0/873223128 shutdown_connections 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc19a540640 1 -- 172.21.15.156:0/873223128 >> 172.21.15.156:0/873223128 conn(0x7fc19409d7a0 msgr2=0x7fc1940d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:43.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc19a540640 1 -- 172.21.15.156:0/873223128 shutdown_connections 2024-03-29T04:09:43.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.581+0000 7fc19a540640 1 -- 172.21.15.156:0/873223128 wait complete. 2024-03-29T04:09:43.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.582+0000 7fc19a540640 1 Processor -- start 2024-03-29T04:09:43.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.582+0000 7fc19a540640 1 -- start start 2024-03-29T04:09:43.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.583+0000 7fc19a540640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc19415cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.583+0000 7fc19a540640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940b4bf0 0x7fc1940cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.583+0000 7fc19a540640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1940ce2a0 0x7fc1941f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.583+0000 7fc19a540640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940d04c0 con 0x7fc17c0039b0 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.583+0000 7fc19a540640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940d0340 con 0x7fc1940ce2a0 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.583+0000 7fc19a540640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc1940d2c20 con 0x7fc1940b4bf0 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc19415cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc19415cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39726/0 (socket says 172.21.15.156:39726) 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 -- 172.21.15.156:0/3107328979 learned_addr learned my addr 172.21.15.156:0/3107328979 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc198add640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1940ce2a0 0x7fc1941f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc1937fe640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940b4bf0 0x7fc1940cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 -- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940b4bf0 msgr2=0x7fc1940cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940b4bf0 0x7fc1940cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 -- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1940ce2a0 msgr2=0x7fc1941f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1940ce2a0 0x7fc1941f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 -- 172.21.15.156:0/3107328979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19409e360 con 0x7fc17c0039b0 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc1937fe640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940b4bf0 0x7fc1940cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc198add640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1940ce2a0 0x7fc1941f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc193fff640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc19415cd90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc18402c3a0 tx=0x7fc184004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184039050 con 0x7fc17c0039b0 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc184045070 con 0x7fc17c0039b0 2024-03-29T04:09:43.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19409e640 con 0x7fc17c0039b0 2024-03-29T04:09:43.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184003a50 con 0x7fc17c0039b0 2024-03-29T04:09:43.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.584+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc19409ec00 con 0x7fc17c0039b0 2024-03-29T04:09:43.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.585+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1940a2480 con 0x7fc17c0039b0 2024-03-29T04:09:43.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.586+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc184003bf0 con 0x7fc17c0039b0 2024-03-29T04:09:43.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.586+0000 7fc190ff9640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc18006c530 0x7fc18006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.586+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc1840ad260 con 0x7fc17c0039b0 2024-03-29T04:09:43.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.586+0000 7fc1937fe640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc18006c530 0x7fc18006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.587+0000 7fc1937fe640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc18006c530 0x7fc18006e9f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc19409f840 tx=0x7fc188008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.590+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc18404e020 con 0x7fc17c0039b0 2024-03-29T04:09:43.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.824+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc1940ce680 con 0x7fc17c0039b0 2024-03-29T04:09:43.804 INFO:tasks.ceph.mon.a.smithi156.stderr:2024-03-29T04:09:43.825+0000 7fc37d67e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T04:09:43.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.826+0000 7fc190ff9640 1 -- 172.21.15.156:0/3107328979 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) ==== 161+0+0 (secure 0 0 0) 0x7fc18407b800 con 0x7fc17c0039b0 2024-03-29T04:09:43.806 INFO:teuthology.orchestra.run.smithi156.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T04:09:43.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc18006c530 msgr2=0x7fc18006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc18006c530 0x7fc18006e9f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc19409f840 tx=0x7fc188008040 comp rx=0 tx=0).stop 2024-03-29T04:09:43.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 msgr2=0x7fc19415cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc19415cd90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc18402c3a0 tx=0x7fc184004730 comp rx=0 tx=0).stop 2024-03-29T04:09:43.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 shutdown_connections 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc18006c530 0x7fc18006e9f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1940ce2a0 0x7fc1941f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc1940b4bf0 0x7fc1940cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 --2- 172.21.15.156:0/3107328979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc17c0039b0 0x7fc19415cd90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.829+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 >> 172.21.15.156:0/3107328979 conn(0x7fc19409d7a0 msgr2=0x7fc1941616b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.830+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 shutdown_connections 2024-03-29T04:09:43.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.830+0000 7fc19a540640 1 -- 172.21.15.156:0/3107328979 wait complete. 2024-03-29T04:09:43.812 INFO:tasks.ceph.mds.c.smithi156.stderr: -3067> 2024-03-29T04:08:58.196+0000 7f92ed70f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T04:09:43.820 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T04:09:43.844 INFO:tasks.ceph.mds.c.smithi156.stdout:starting mds.c at 2024-03-29T04:09:43.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.945+0000 7fa6f7500640 1 Processor -- start 2024-03-29T04:09:43.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.945+0000 7fa6f7500640 1 -- start start 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.946+0000 7fa6f7500640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa6f01415e0 0x7fa6f0144460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.946+0000 7fa6f7500640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f0141f60 con 0x7fa6f01449a0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.946+0000 7fa6f7500640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f0142240 con 0x7fa6f0143ec0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.946+0000 7fa6f7500640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f0141d10 con 0x7fa6f01415e0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.946+0000 7fa6f4a9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa6f01449a0 0x7fa6f00b54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44740/0 (socket says 172.21.15.156:44740) 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.946+0000 7fa6f4a9b640 1 -- 172.21.15.156:0/1093133120 learned_addr learned my addr 172.21.15.156:0/1093133120 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1093133120 >> v2:172.21.15.203:3300/0 conn(0x7fa6f01415e0 0x7fa6f0144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2413272469 0 0) 0x7fa6f0141f60 con 0x7fa6f01449a0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6dc003520 con 0x7fa6f01449a0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3490317852 0 0) 0x7fa6dc003520 con 0x7fa6f01449a0 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 >> v2:172.21.15.203:3300/0 conn(0x7fa6f01415e0 msgr2=0x7fa6f0144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 --2- 172.21.15.156:0/1093133120 >> v2:172.21.15.203:3300/0 conn(0x7fa6f01415e0 0x7fa6f0144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 >> v1:172.21.15.203:6789/0 conn(0x7fa6f0143ec0 legacy=0x7fa6f0071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f0142520 con 0x7fa6f01449a0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fa6e0001670 con 0x7fa6f01449a0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 >> v1:172.21.15.156:6789/0 conn(0x7fa6f01449a0 legacy=0x7fa6f00b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6dc0039b0 0x7fa6dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc004800 0x7fa6dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6dc0052e0 0x7fa6dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0141d10 con 0x7fa6dc0039b0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc00c1b0 con 0x7fa6dc004800 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa6dc00c3b0 con 0x7fa6dc0052e0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6dc0039b0 0x7fa6dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f4a9b640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc004800 0x7fa6dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f5a9d640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6dc0052e0 0x7fa6dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 -- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6dc0052e0 msgr2=0x7fa6dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6dc0052e0 0x7fa6dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 -- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc004800 msgr2=0x7fa6dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc004800 0x7fa6dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f5a9d640 1 -- 172.21.15.156:0/1093133120 reap_dead start 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.947+0000 7fa6f529c640 1 -- 172.21.15.156:0/1093133120 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc003520 con 0x7fa6dc0039b0 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.948+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6dc0039b0 0x7fa6dc003dd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009f90 tx=0x7fa6dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4944b61fd710155b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.948+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec039070 con 0x7fa6dc0039b0 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.948+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6ec02cbd0 con 0x7fa6dc0039b0 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.948+0000 7fa6e7fff640 1 -- 172.21.15.156:0/1093133120 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec009c10 con 0x7fa6dc0039b0 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 -- 172.21.15.156:0/1093133120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6dc0039b0 msgr2=0x7fa6dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6dc0039b0 0x7fa6dc003dd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009f90 tx=0x7fa6dc005830 comp rx=0 tx=0).stop 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 -- 172.21.15.156:0/1093133120 shutdown_connections 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1093133120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6dc0039b0 0x7fa6dc003dd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 -- 172.21.15.156:0/1093133120 >> 172.21.15.156:0/1093133120 conn(0x7fa6f00b3eb0 msgr2=0x7fa6f00b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:43.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 -- 172.21.15.156:0/1093133120 shutdown_connections 2024-03-29T04:09:43.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.949+0000 7fa6f7500640 1 -- 172.21.15.156:0/1093133120 wait complete. 2024-03-29T04:09:43.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.950+0000 7fa6f7500640 1 Processor -- start 2024-03-29T04:09:43.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.950+0000 7fa6f7500640 1 -- start start 2024-03-29T04:09:43.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc0039b0 0x7fa6f0147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 0x7fa6f01459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6f0145f10 0x7fa6f0146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f006c6c0 con 0x7fa6f013bcb0 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01419c0 con 0x7fa6dc0039b0 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f006c840 con 0x7fa6f0145f10 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 0x7fa6f01459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 0x7fa6f01459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39776/0 (socket says 172.21.15.156:39776) 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 -- 172.21.15.156:0/1634637528 learned_addr learned my addr 172.21.15.156:0/1634637528 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f5a9d640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6f0145f10 0x7fa6f0146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc0039b0 0x7fa6f0147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 -- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6f0145f10 msgr2=0x7fa6f0146580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6f0145f10 0x7fa6f0146580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 -- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc0039b0 msgr2=0x7fa6f0147120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc0039b0 0x7fa6f0147120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 -- 172.21.15.156:0/1634637528 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f0147660 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f5a9d640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6f0145f10 0x7fa6f0146580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f4a9b640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 0x7fa6f01459d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0007310 tx=0x7fa6e00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0001430 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e0018070 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f0147930 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.951+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e000fba0 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.952+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f01451d0 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.952+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f0070e10 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.953+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa6e000fd40 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.953+0000 7fa6e5ffb640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6d406c560 0x7fa6d406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.953+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6d406c560 0x7fa6d406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.953+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa6e008b740 con 0x7fa6f013bcb0 2024-03-29T04:09:43.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.954+0000 7fa6f529c640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6d406c560 0x7fa6d406ea20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec036000 tx=0x7fa6ec02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:43.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:43.958+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa6e0059c90 con 0x7fa6f013bcb0 2024-03-29T04:09:44.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.171+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa6f004f9d0 con 0x7fa6f013bcb0 2024-03-29T04:09:44.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.828+0000 7fa6e5ffb640 1 -- 172.21.15.156:0/1634637528 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) ==== 104+0+0 (secure 0 0 0) 0x7fa6e0059590 con 0x7fa6f013bcb0 2024-03-29T04:09:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6d406c560 msgr2=0x7fa6d406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6d406c560 0x7fa6d406ea20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec036000 tx=0x7fa6ec02e000 comp rx=0 tx=0).stop 2024-03-29T04:09:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 msgr2=0x7fa6f01459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 0x7fa6f01459d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0007310 tx=0x7fa6e00077d0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 shutdown_connections 2024-03-29T04:09:44.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6d406c560 0x7fa6d406ea20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa6f0145f10 0x7fa6f0146580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6f013bcb0 0x7fa6f01459d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 --2- 172.21.15.156:0/1634637528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6dc0039b0 0x7fa6f0147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 >> 172.21.15.156:0/1634637528 conn(0x7fa6f00b3eb0 msgr2=0x7fa6f00b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:44.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 shutdown_connections 2024-03-29T04:09:44.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.830+0000 7fa6f7500640 1 -- 172.21.15.156:0/1634637528 wait complete. 2024-03-29T04:09:44.822 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T04:09:44.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.992+0000 7f8adab2f640 1 Processor -- start 2024-03-29T04:09:44.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.992+0000 7f8adab2f640 1 -- start start 2024-03-29T04:09:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.992+0000 7f8adab2f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8ad40d0d20 0x7f8ad4160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.992+0000 7f8adab2f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad409b7c0 con 0x7f8ad406f9f0 2024-03-29T04:09:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.992+0000 7f8adab2f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad40ccd90 con 0x7f8ad40d0940 2024-03-29T04:09:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.992+0000 7f8adab2f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad40ceb20 con 0x7f8ad40d0d20 2024-03-29T04:09:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8ad40d0d20 0x7f8ad4160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:44.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad90cc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8ad406f9f0 0x7f8ad406fe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44754/0 (socket says 172.21.15.156:44754) 2024-03-29T04:09:44.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad90cc640 1 -- 172.21.15.156:0/3699307780 learned_addr learned my addr 172.21.15.156:0/3699307780 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:44.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad37fe640 1 -- 172.21.15.156:0/3699307780 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432685444 0 0) 0x7f8ad409b7c0 con 0x7f8ad406f9f0 2024-03-29T04:09:44.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad37fe640 1 -- 172.21.15.156:0/3699307780 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ac4003520 con 0x7f8ad406f9f0 2024-03-29T04:09:44.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad3fff640 1 -- 172.21.15.156:0/3699307780 >> v1:172.21.15.203:6789/0 conn(0x7f8ad40d0940 legacy=0x7f8ad4160530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:44.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad3fff640 1 -- 172.21.15.156:0/3699307780 >> v1:172.21.15.156:6789/0 conn(0x7f8ad406f9f0 legacy=0x7f8ad406fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:44.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.993+0000 7f8ad3fff640 1 -- 172.21.15.156:0/3699307780 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40cd030 con 0x7f8ad40d0d20 2024-03-29T04:09:44.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8ad3fff640 1 --2- 172.21.15.156:0/3699307780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0d20 0x7f8ad4160c40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00097a0 tx=0x7f8ac002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb44fed7b09691d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:44.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8ad37fe640 1 -- 172.21.15.156:0/3699307780 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0038070 con 0x7f8ad40d0d20 2024-03-29T04:09:44.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8ad37fe640 1 -- 172.21.15.156:0/3699307780 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac0031700 con 0x7f8ad40d0d20 2024-03-29T04:09:44.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8ad37fe640 1 -- 172.21.15.156:0/3699307780 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0031ad0 con 0x7f8ad40d0d20 2024-03-29T04:09:44.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 -- 172.21.15.156:0/3699307780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0d20 msgr2=0x7f8ad4160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:44.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 --2- 172.21.15.156:0/3699307780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0d20 0x7f8ad4160c40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00097a0 tx=0x7f8ac002b7a0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 -- 172.21.15.156:0/3699307780 shutdown_connections 2024-03-29T04:09:44.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 --2- 172.21.15.156:0/3699307780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0d20 0x7f8ad4160c40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 -- 172.21.15.156:0/3699307780 >> 172.21.15.156:0/3699307780 conn(0x7f8ad40d2280 msgr2=0x7f8ad40d2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:44.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 -- 172.21.15.156:0/3699307780 shutdown_connections 2024-03-29T04:09:44.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.994+0000 7f8adab2f640 1 -- 172.21.15.156:0/3699307780 wait complete. 2024-03-29T04:09:44.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.995+0000 7f8adab2f640 1 Processor -- start 2024-03-29T04:09:44.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 -- start start 2024-03-29T04:09:44.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ad406f9f0 0x7f8ad41f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:44.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 0x7f8ad41f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:44.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ad40d0d20 0x7f8ad40b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:44.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40af0d0 con 0x7f8ad40d0940 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4071570 con 0x7f8ad40d0d20 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.996+0000 7f8adab2f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad40af250 con 0x7f8ad406f9f0 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 0x7f8ad41f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 0x7f8ad41f1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39804/0 (socket says 172.21.15.156:39804) 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 -- 172.21.15.156:0/1538147709 learned_addr learned my addr 172.21.15.156:0/1538147709 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad88cb640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ad406f9f0 0x7f8ad41f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad90cc640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ad40d0d20 0x7f8ad40b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 -- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ad406f9f0 msgr2=0x7f8ad41f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ad406f9f0 0x7f8ad41f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 -- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ad40d0d20 msgr2=0x7f8ad40b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ad40d0d20 0x7f8ad40b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 -- 172.21.15.156:0/1538147709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad41f2200 con 0x7f8ad40d0940 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad3fff640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 0x7f8ad41f1cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0043070 con 0x7f8ad40d0940 2024-03-29T04:09:44.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac003f890 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad41f32b0 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0038040 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.997+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad40b63d0 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.998+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad409bad0 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.999+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8ac0046070 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.999+0000 7f8ad17fa640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ab806c530 0x7f8ab806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.999+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8ac00ad5c0 con 0x7f8ad40d0940 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.999+0000 7f8ad88cb640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ab806c530 0x7f8ab806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:44.999+0000 7f8ad88cb640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ab806c530 0x7f8ab806e9f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8007fd0 tx=0x7f8ac8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:44.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.003+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8ad409bad0 con 0x7f8ad40d0940 2024-03-29T04:09:45.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.179+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8ad40cf030 con 0x7f8ad40d0940 2024-03-29T04:09:45.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.972+0000 7f8ad17fa640 1 -- 172.21.15.156:0/1538147709 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) ==== 181+0+0 (secure 0 0 0) 0x7f8ac0030070 con 0x7f8ad40d0940 2024-03-29T04:09:45.952 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_metadata' removed 2024-03-29T04:09:45.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ab806c530 msgr2=0x7f8ab806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:45.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ab806c530 0x7f8ab806e9f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8007fd0 tx=0x7f8ac8009040 comp rx=0 tx=0).stop 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 msgr2=0x7f8ad41f1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 0x7f8ad41f1cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac00046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 shutdown_connections 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8ab806c530 0x7f8ab806e9f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ad40d0d20 0x7f8ad40b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad40d0940 0x7f8ad41f1cc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 --2- 172.21.15.156:0/1538147709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ad406f9f0 0x7f8ad41f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 >> 172.21.15.156:0/1538147709 conn(0x7f8ad40d2280 msgr2=0x7f8ad40c8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 shutdown_connections 2024-03-29T04:09:45.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:45.976+0000 7f8adab2f640 1 -- 172.21.15.156:0/1538147709 wait complete. 2024-03-29T04:09:45.969 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T04:09:46.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.097+0000 7f2ef8a88640 1 Processor -- start 2024-03-29T04:09:46.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.097+0000 7f2ef8a88640 1 -- start start 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef8a88640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2ef4160780 0x7f2ef40cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef8a88640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ef40d0360 con 0x7f2ef40b7f50 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef8a88640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ef40d0640 con 0x7f2ef40cfe60 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef8a88640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2ef40d0f70 con 0x7f2ef4160780 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef1d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2ef4160780 0x7f2ef40cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef259c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2ef40b7f50 0x7f2ef4160070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44768/0 (socket says 172.21.15.156:44768) 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.098+0000 7f2ef259c640 1 -- 172.21.15.156:0/820219129 learned_addr learned my addr 172.21.15.156:0/820219129 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:46.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.099+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/820219129 >> v1:172.21.15.203:6789/0 conn(0x7f2ef40cfe60 legacy=0x7f2ef415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:46.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.099+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/820219129 >> v1:172.21.15.156:6789/0 conn(0x7f2ef40b7f50 legacy=0x7f2ef4160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:46.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.099+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/820219129 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef40d0920 con 0x7f2ef4160780 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.099+0000 7f2ef1d9b640 1 --2- 172.21.15.156:0/820219129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef4160780 0x7f2ef40cdd70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d31f1319ef621b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.099+0000 7f2ef159a640 1 -- 172.21.15.156:0/820219129 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8035020 con 0x7f2ef4160780 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef159a640 1 -- 172.21.15.156:0/820219129 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ee8031830 con 0x7f2ef4160780 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef159a640 1 -- 172.21.15.156:0/820219129 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8031c00 con 0x7f2ef4160780 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 -- 172.21.15.156:0/820219129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef4160780 msgr2=0x7f2ef40cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/820219129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef4160780 0x7f2ef40cdd70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 -- 172.21.15.156:0/820219129 shutdown_connections 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/820219129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef4160780 0x7f2ef40cdd70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 -- 172.21.15.156:0/820219129 >> 172.21.15.156:0/820219129 conn(0x7f2ef40d2290 msgr2=0x7f2ef40d26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 -- 172.21.15.156:0/820219129 shutdown_connections 2024-03-29T04:09:46.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.100+0000 7f2ef8a88640 1 -- 172.21.15.156:0/820219129 wait complete. 2024-03-29T04:09:46.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.101+0000 7f2ef8a88640 1 Processor -- start 2024-03-29T04:09:46.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.101+0000 7f2ef8a88640 1 -- start start 2024-03-29T04:09:46.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef8a88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef40b7f50 0x7f2ef4154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:46.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef8a88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 0x7f2ef40c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:46.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef8a88640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ef4160780 0x7f2ef4070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:46.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef8a88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef40cd1b0 con 0x7f2ef40cfe60 2024-03-29T04:09:46.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef8a88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef40cd030 con 0x7f2ef40b7f50 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef8a88640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2ef40b6200 con 0x7f2ef4160780 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 0x7f2ef40c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 0x7f2ef40c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39826/0 (socket says 172.21.15.156:39826) 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.102+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/1417553993 learned_addr learned my addr 172.21.15.156:0/1417553993 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef259c640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef40b7f50 0x7f2ef4154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ef4160780 msgr2=0x7f2ef4070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef2d9d640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ef4160780 0x7f2ef4070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef1d9b640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ef4160780 0x7f2ef4070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef40b7f50 msgr2=0x7f2ef4154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef1d9b640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef40b7f50 0x7f2ef4154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef1d9b640 1 -- 172.21.15.156:0/1417553993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef4154940 con 0x7f2ef40cfe60 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef259c640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef40b7f50 0x7f2ef4154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:46.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef1d9b640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 0x7f2ef40c6e10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8034040 tx=0x7f2ee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:46.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee803b9c0 con 0x7f2ef40cfe60 2024-03-29T04:09:46.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ee803bba0 con 0x7f2ef40cfe60 2024-03-29T04:09:46.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef40c7640 con 0x7f2ef40cfe60 2024-03-29T04:09:46.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80453f0 con 0x7f2ef40cfe60 2024-03-29T04:09:46.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.103+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef40c7c00 con 0x7f2ef40cfe60 2024-03-29T04:09:46.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.104+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef4068610 con 0x7f2ef40cfe60 2024-03-29T04:09:46.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.104+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2ee804c060 con 0x7f2ef40cfe60 2024-03-29T04:09:46.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.105+0000 7f2ee2ffd640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2ed406c560 0x7f2ed406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:46.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.105+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5473+0+0 (secure 0 0 0) 0x7f2ee80ad260 con 0x7f2ef40cfe60 2024-03-29T04:09:46.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.105+0000 7f2ef259c640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2ed406c560 0x7f2ed406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:46.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.105+0000 7f2ef259c640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2ed406c560 0x7f2ed406ea20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40012e0 tx=0x7f2ee4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:46.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.109+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2ef4154940 con 0x7f2ef40cfe60 2024-03-29T04:09:46.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.283+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2ef40b5fc0 con 0x7f2ef40cfe60 2024-03-29T04:09:46.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.980+0000 7f2ee2ffd640 1 -- 172.21.15.156:0/1417553993 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) ==== 169+0+0 (secure 0 0 0) 0x7f2ee8040070 con 0x7f2ef40cfe60 2024-03-29T04:09:46.959 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_data' removed 2024-03-29T04:09:46.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.983+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2ed406c560 msgr2=0x7f2ed406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:46.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.983+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2ed406c560 0x7f2ed406ea20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40012e0 tx=0x7f2ee4009040 comp rx=0 tx=0).stop 2024-03-29T04:09:46.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.983+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 msgr2=0x7f2ef40c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:46.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.983+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 0x7f2ef40c6e10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8034040 tx=0x7f2ee80046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 shutdown_connections 2024-03-29T04:09:46.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2ed406c560 0x7f2ed406ea20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ef4160780 0x7f2ef4070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ef40cfe60 0x7f2ef40c6e10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 --2- 172.21.15.156:0/1417553993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef40b7f50 0x7f2ef4154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:46.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 >> 172.21.15.156:0/1417553993 conn(0x7f2ef40d2290 msgr2=0x7f2ef40ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:46.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 shutdown_connections 2024-03-29T04:09:46.966 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:46.984+0000 7f2ef8a88640 1 -- 172.21.15.156:0/1417553993 wait complete. 2024-03-29T04:09:46.977 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T04:09:46.978 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T04:09:46.978 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T04:09:46.978 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:46.978 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T04:09:46.978 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T04:09:46.978 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T04:09:46.979 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:46.979 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T04:09:46.979 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T04:09:46.979 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T04:09:46.979 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:46.980 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T04:09:46.980 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T04:09:46.980 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T04:09:46.980 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:09:46.988 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T04:09:46.988 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T04:09:46.989 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T04:09:46.989 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T04:09:46.991 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T04:09:46.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T04:09:46.992 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T04:09:46.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T04:09:47.031 INFO:tasks.ceph.mds.a:Started 2024-03-29T04:09:47.031 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:47.032 INFO:tasks.ceph.mds.c:Started 2024-03-29T04:09:47.033 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:47.034 INFO:tasks.ceph.mds.b:Started 2024-03-29T04:09:47.034 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:47.035 INFO:tasks.ceph.mds.d:Started 2024-03-29T04:09:47.035 DEBUG:teuthology.parallel:result is None 2024-03-29T04:09:47.035 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T04:09:47.121 INFO:tasks.ceph.mds.c.smithi156.stdout:starting mds.c at 2024-03-29T04:09:47.122 INFO:tasks.ceph.mds.a.smithi156.stdout:starting mds.a at 2024-03-29T04:09:47.125 INFO:tasks.ceph.mds.d.smithi203.stdout:starting mds.d at 2024-03-29T04:09:47.125 INFO:tasks.ceph.mds.b.smithi203.stdout:starting mds.b at 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.173+0000 7f98a19d4640 1 Processor -- start 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.173+0000 7f98a19d4640 1 -- start start 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f98a19d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c15e920 0x7f989c0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f98a19d4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f989c0d2120 0x7f989c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f98a19d4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f989c06cb10 0x7f989c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f98a19d4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f989c0cf7a0 con 0x7f989c06cb10 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f98a19d4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f989c0cf190 con 0x7f989c0d2120 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f98a19d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f989c0d3080 con 0x7f989c15e920 2024-03-29T04:09:47.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f989b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f989c06cb10 0x7f989c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f989b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f989c06cb10 0x7f989c0cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39854/0 (socket says 172.21.15.156:39854) 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f989b7fe640 1 -- 172.21.15.156:0/4060350521 learned_addr learned my addr 172.21.15.156:0/4060350521 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f989affd640 1 --2- 172.21.15.156:0/4060350521 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c15e920 0x7f989c0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f989a7fc640 1 --2- 172.21.15.156:0/4060350521 >> v2:172.21.15.203:3300/0 conn(0x7f989c0d2120 0x7f989c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.174+0000 7f989b7fe640 1 -- 172.21.15.156:0/4060350521 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c15e920 msgr2=0x7f989c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f989b7fe640 1 --2- 172.21.15.156:0/4060350521 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c15e920 0x7f989c0d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f989b7fe640 1 -- 172.21.15.156:0/4060350521 >> v2:172.21.15.203:3300/0 conn(0x7f989c0d2120 msgr2=0x7f989c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f989b7fe640 1 --2- 172.21.15.156:0/4060350521 >> v2:172.21.15.203:3300/0 conn(0x7f989c0d2120 0x7f989c06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f989b7fe640 1 -- 172.21.15.156:0/4060350521 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989c0cf430 con 0x7f989c06cb10 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f989a7fc640 1 --2- 172.21.15.156:0/4060350521 >> v2:172.21.15.203:3300/0 conn(0x7f989c0d2120 0x7f989c06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f989b7fe640 1 --2- 172.21.15.156:0/4060350521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c0cfc30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f989c08c930 tx=0x7f989402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57dd34d79800430 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f9899ffb640 1 -- 172.21.15.156:0/4060350521 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9894035020 con 0x7f989c06cb10 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f9899ffb640 1 -- 172.21.15.156:0/4060350521 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9894031870 con 0x7f989c06cb10 2024-03-29T04:09:47.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.175+0000 7f9899ffb640 1 -- 172.21.15.156:0/4060350521 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9894031c40 con 0x7f989c06cb10 2024-03-29T04:09:47.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 -- 172.21.15.156:0/4060350521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 msgr2=0x7f989c0cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 --2- 172.21.15.156:0/4060350521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c0cfc30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f989c08c930 tx=0x7f989402b910 comp rx=0 tx=0).stop 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 -- 172.21.15.156:0/4060350521 shutdown_connections 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 --2- 172.21.15.156:0/4060350521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c0cfc30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 --2- 172.21.15.156:0/4060350521 >> v2:172.21.15.203:3300/0 conn(0x7f989c0d2120 0x7f989c06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 --2- 172.21.15.156:0/4060350521 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c15e920 0x7f989c0d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 -- 172.21.15.156:0/4060350521 >> 172.21.15.156:0/4060350521 conn(0x7f989c09d7a0 msgr2=0x7f989c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 -- 172.21.15.156:0/4060350521 shutdown_connections 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.176+0000 7f98a19d4640 1 -- 172.21.15.156:0/4060350521 wait complete. 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.177+0000 7f98a19d4640 1 Processor -- start 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.177+0000 7f98a19d4640 1 -- start start 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f98a19d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f98a19d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c0d2120 0x7f989c0b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f98a19d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f989c15e920 0x7f989c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f98a19d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f989c0d2da0 con 0x7f989c06cb10 2024-03-29T04:09:47.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f98a19d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f989c0d2c20 con 0x7f989c15e920 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f98a19d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f989c0d0170 con 0x7f989c0d2120 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f989affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f989affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39866/0 (socket says 172.21.15.156:39866) 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f989affd640 1 -- 172.21.15.156:0/812092549 learned_addr learned my addr 172.21.15.156:0/812092549 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.178+0000 7f989b7fe640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f989c15e920 0x7f989c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989a7fc640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c0d2120 0x7f989c0b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989affd640 1 -- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c0d2120 msgr2=0x7f989c0b8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989affd640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c0d2120 0x7f989c0b8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989affd640 1 -- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f989c15e920 msgr2=0x7f989c0b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989affd640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f989c15e920 0x7f989c0b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989affd640 1 -- 172.21.15.156:0/812092549 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989c0b8f40 con 0x7f989c06cb10 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989b7fe640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f989c15e920 0x7f989c0b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f989affd640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c159110 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f988c007e40 tx=0x7f988c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c014620 con 0x7f989c06cb10 2024-03-29T04:09:47.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f988c019070 con 0x7f989c06cb10 2024-03-29T04:09:47.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f989c0b9220 con 0x7f989c06cb10 2024-03-29T04:09:47.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c014a00 con 0x7f989c06cb10 2024-03-29T04:09:47.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.179+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f989c0b97e0 con 0x7f989c06cb10 2024-03-29T04:09:47.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.180+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f989c0a2480 con 0x7f989c06cb10 2024-03-29T04:09:47.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.181+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f988c014ba0 con 0x7f989c06cb10 2024-03-29T04:09:47.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.181+0000 7f9883fff640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f988806c530 0x7f988806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:47.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.181+0000 7f989a7fc640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f988806c530 0x7f988806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:47.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.181+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4943+0+0 (secure 0 0 0) 0x7f988c08bc90 con 0x7f989c06cb10 2024-03-29T04:09:47.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.182+0000 7f989a7fc640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f988806c530 0x7f988806e9f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:47.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.186+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f988c05a5b0 con 0x7f989c06cb10 2024-03-29T04:09:47.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.371+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f989c0cfcd0 con 0x7f989c06cb10 2024-03-29T04:09:47.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.992+0000 7f9883fff640 1 -- 172.21.15.156:0/812092549 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) ==== 97+0+0 (secure 0 0 0) 0x7f988c059eb0 con 0x7f989c06cb10 2024-03-29T04:09:47.972 INFO:teuthology.orchestra.run.smithi156.stderr: removed all blocklist entries 2024-03-29T04:09:47.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f988806c530 msgr2=0x7f988806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:47.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f988806c530 0x7f988806e9f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890008040 comp rx=0 tx=0).stop 2024-03-29T04:09:47.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 msgr2=0x7f989c159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c159110 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f988c007e40 tx=0x7f988c00e4b0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 shutdown_connections 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f988806c530 0x7f988806e9f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f989c15e920 0x7f989c0b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989c0d2120 0x7f989c0b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 --2- 172.21.15.156:0/812092549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f989c06cb10 0x7f989c159110 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 >> 172.21.15.156:0/812092549 conn(0x7f989c09d7a0 msgr2=0x7f989c0d4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 shutdown_connections 2024-03-29T04:09:47.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:47.996+0000 7f98a19d4640 1 -- 172.21.15.156:0/812092549 wait complete. 2024-03-29T04:09:47.988 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T04:09:48.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.110+0000 7f561061b640 1 Processor -- start 2024-03-29T04:09:48.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.110+0000 7f561061b640 1 -- start start 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f561061b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f560815c270 0x7f5608142690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f561061b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815c650 0x7f56080ad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f561061b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56080b30f0 con 0x7f560815cad0 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f561061b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f5608142bd0 con 0x7f560815c270 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f561061b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f560815d020 con 0x7f560815c650 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f560815c270 0x7f5608142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f560815c270 0x7f5608142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39868/0 (socket says 172.21.15.156:39868) 2024-03-29T04:09:48.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 -- 172.21.15.156:0/4278320256 learned_addr learned my addr 172.21.15.156:0/4278320256 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 -- 172.21.15.156:0/4278320256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815c650 msgr2=0x7f56080ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 --2- 172.21.15.156:0/4278320256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815c650 0x7f56080ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 -- 172.21.15.156:0/4278320256 >> v1:172.21.15.203:6789/0 conn(0x7f560815cad0 legacy=0x7f560815ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:48.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.111+0000 7f560e3b7640 1 -- 172.21.15.156:0/4278320256 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560815d300 con 0x7f560815c270 2024-03-29T04:09:48.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.112+0000 7f560e3b7640 1 --2- 172.21.15.156:0/4278320256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f5608142690 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5604009870 tx=0x7f560402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a6ac93b87f5da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.112+0000 7f560d3b5640 1 -- 172.21.15.156:0/4278320256 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56040317c0 con 0x7f560815c270 2024-03-29T04:09:48.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.112+0000 7f560d3b5640 1 -- 172.21.15.156:0/4278320256 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5604037070 con 0x7f560815c270 2024-03-29T04:09:48.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.112+0000 7f560d3b5640 1 -- 172.21.15.156:0/4278320256 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604031b80 con 0x7f560815c270 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 -- 172.21.15.156:0/4278320256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 msgr2=0x7f5608142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 --2- 172.21.15.156:0/4278320256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f5608142690 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5604009870 tx=0x7f560402b860 comp rx=0 tx=0).stop 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 -- 172.21.15.156:0/4278320256 shutdown_connections 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 --2- 172.21.15.156:0/4278320256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815c650 0x7f56080ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 --2- 172.21.15.156:0/4278320256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f5608142690 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 -- 172.21.15.156:0/4278320256 >> 172.21.15.156:0/4278320256 conn(0x7f56080b1d50 msgr2=0x7f56080b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 -- 172.21.15.156:0/4278320256 shutdown_connections 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.113+0000 7f561061b640 1 -- 172.21.15.156:0/4278320256 wait complete. 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.114+0000 7f561061b640 1 Processor -- start 2024-03-29T04:09:48.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.114+0000 7f561061b640 1 -- start start 2024-03-29T04:09:48.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f561061b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f56081f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f561061b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560815c650 0x7f5608152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f561061b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815cad0 0x7f5608144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f561061b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56080b45e0 con 0x7f560815c270 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f561061b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f560815d8c0 con 0x7f560815c650 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f561061b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f56080b4760 con 0x7f560815cad0 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f56081f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f56081f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39882/0 (socket says 172.21.15.156:39882) 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 -- 172.21.15.156:0/1000718266 learned_addr learned my addr 172.21.15.156:0/1000718266 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560ebb8640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815cad0 0x7f5608144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560dbb6640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560815c650 0x7f5608152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 -- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815cad0 msgr2=0x7f5608144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815cad0 0x7f5608144340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 -- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560815c650 msgr2=0x7f5608152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560815c650 0x7f5608152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 -- 172.21.15.156:0/1000718266 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5608154580 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.115+0000 7f560e3b7640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f56081f2410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5604009870 tx=0x7f56040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.116+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604032860 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.116+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5604032a00 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.116+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5608154860 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.116+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604032df0 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.116+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5608155100 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.116+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5608154a30 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.117+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f560404c060 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.117+0000 7f55feffd640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f55ec06c530 0x7f55ec06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.117+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f56040acce0 con 0x7f560815c270 2024-03-29T04:09:48.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.117+0000 7f560dbb6640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f55ec06c530 0x7f55ec06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.099 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.118+0000 7f560dbb6640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f55ec06c530 0x7f55ec06e9f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5608151520 tx=0x7f55f8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.101 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.122+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5604040070 con 0x7f560815c270 2024-03-29T04:09:48.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.363+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f560815dae0 con 0x7f560815c270 2024-03-29T04:09:48.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.363+0000 7f55feffd640 1 -- 172.21.15.156:0/1000718266 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f560815dae0 con 0x7f560815c270 2024-03-29T04:09:48.343 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:48.343 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:09:48.343 INFO:teuthology.orchestra.run.smithi156.stdout: "auth_dump": [ 2024-03-29T04:09:48.343 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.343 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "mds.a", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZmizBpJxAAE+UIT0JKBcd+kqBm8OXqJw==", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow *", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow *", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "mds.b", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZmzh3TMxAAZG099AMS07J8m9Ou1su9Yg==", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.344 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow *", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow *", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "mds.c", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZmmynKLhAA381EIczFudxSPbO5dsgymw==", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow *", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow *", 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.345 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "mds.d", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZmuIA3OxAAVTf4wVco/HrNU6fR69Nf/Q==", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow *", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow *", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.0", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCfPgZmWTSSJRAAMdLo56etlMy+UbuyTQxQ6g==", 2024-03-29T04:09:48.346 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.1", 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCiPgZmvOckJBAA9sSR5OsAa3EEUuBvADm2AQ==", 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.347 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.2", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQClPgZm1tyIIhAApvrDTh0Z96t6YlGcKltc2g==", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.3", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCoPgZmXJjCIBAA2Mew2foyDIQLWozE9B/rCg==", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.4", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCuPgZmQh6oJBAAcylK7X8CI9jQn6Tq7v80hw==", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.5", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCxPgZm3rGYIxAA690kx1Ijuzov0SoiOYtLjQ==", 2024-03-29T04:09:48.349 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.6", 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC0PgZm5PWsIRAAO1bIBx89FMZwhLquciqsvg==", 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.350 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "osd.7", 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC3PgZmLzxsHxAAJfLHVYR7HNXAqJav1bKs5g==", 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow profile osd", 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile osd", 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.0", 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCePgZmw7rsAxAA9fTOYwz/xRpNNRZVKRBg3w==", 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.351 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow r", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow rw", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow rwx" 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.1", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCePgZmTQIwDRAAupi7/jFYwobM49T0Gl9F1g==", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow r", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow rw", 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow rwx" 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.352 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.2", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCePgZmwDS+CRAAW5ocjkKOFopCBTMtmRnsUQ==", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow r", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow rw", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow rwx" 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.3", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCePgZm0dIAExAAPMAWLLow0ngSQ0lNsDl9rA==", 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.353 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow r", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow rw", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow rwx" 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.admin", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZmPm72CRAAG5oTFGdl2Rp9W2fIWkLy+Q==", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow *", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "mgr": "allow *", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow *", 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.354 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.bootstrap-mds", 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC9PgZmioeZGBAAwnLTVwcEemr5TXzb1O6joQ==", 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile bootstrap-mds" 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.bootstrap-mgr", 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC9PgZmIZCZGBAADgKNu7tPs233zjSzHEYaog==", 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-29T04:09:48.355 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.bootstrap-osd", 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC9PgZmDJeZGBAAM3AuCPHRlQcIP3Z95mBFZQ==", 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile bootstrap-osd" 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.bootstrap-rbd", 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC9PgZmup2ZGBAAwRbz4e3M9+zJL/5r9D+pIQ==", 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-29T04:09:48.356 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC9PgZmlqSZGBAAdbxu8VJV6bPdhBnS6qkQSA==", 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "client.bootstrap-rgw", 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQC9PgZmoKuZGBAATrI6/9bNPEK1cDR5EWx8Jg==", 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.357 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "mgr.x", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZmEsv5IxAA7BsWzOOHkozAsY0Z+Jcwrg==", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow *", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile mgr", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "entity": "mgr.y", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "AQCdPgZm50w1HxAAw/V6YkFKxVtZwbu6zOW8kQ==", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": { 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": "allow *", 2024-03-29T04:09:48.358 INFO:teuthology.orchestra.run.smithi156.stdout: "mon": "allow profile mgr", 2024-03-29T04:09:48.359 INFO:teuthology.orchestra.run.smithi156.stdout: "osd": "allow *" 2024-03-29T04:09:48.359 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.359 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:48.359 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-03-29T04:09:48.359 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:09:48.360 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f55ec06c530 msgr2=0x7f55ec06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.360 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f55ec06c530 0x7f55ec06e9f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5608151520 tx=0x7f55f8009060 comp rx=0 tx=0).stop 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 msgr2=0x7f56081f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f56081f2410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5604009870 tx=0x7f56040046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 shutdown_connections 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f55ec06c530 0x7f55ec06e9f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f560815cad0 0x7f5608144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560815c650 0x7f5608152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 --2- 172.21.15.156:0/1000718266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f560815c270 0x7f56081f2410 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.365+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 >> 172.21.15.156:0/1000718266 conn(0x7f56080b1d50 msgr2=0x7f56080b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.366+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 shutdown_connections 2024-03-29T04:09:48.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.366+0000 7f561061b640 1 -- 172.21.15.156:0/1000718266 wait complete. 2024-03-29T04:09:48.362 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-29T04:09:48.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.483+0000 7f4d3027b640 1 Processor -- start 2024-03-29T04:09:48.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.483+0000 7f4d3027b640 1 -- start start 2024-03-29T04:09:48.464 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d3027b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4d2815c390 0x7f4d2806d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.464 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d3027b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2815b8d0 con 0x7f4d280d0350 2024-03-29T04:09:48.464 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d3027b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2806e120 con 0x7f4d2806dba0 2024-03-29T04:09:48.464 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d3027b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4d2815bf00 con 0x7f4d2815c390 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d2e017640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4d280d0350 0x7f4d280d0730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44792/0 (socket says 172.21.15.156:44792) 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d2e017640 1 -- 172.21.15.156:0/649849122 learned_addr learned my addr 172.21.15.156:0/649849122 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.484+0000 7f4d2d816640 1 --2- 172.21.15.156:0/649849122 >> v2:172.21.15.203:3300/0 conn(0x7f4d2815c390 0x7f4d2806d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480710473 0 0) 0x7f4d2815b8d0 con 0x7f4d280d0350 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d14003520 con 0x7f4d280d0350 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1245916039 0 0) 0x7f4d2806e120 con 0x7f4d2806dba0 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d2815b8d0 con 0x7f4d2806dba0 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d816640 1 -- 172.21.15.156:0/649849122 >> v1:172.21.15.203:6789/0 conn(0x7f4d2806dba0 legacy=0x7f4d2815eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d816640 1 -- 172.21.15.156:0/649849122 >> v1:172.21.15.156:6789/0 conn(0x7f4d280d0350 legacy=0x7f4d280d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.485+0000 7f4d2d816640 1 -- 172.21.15.156:0/649849122 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2806e400 con 0x7f4d2815c390 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d2d816640 1 --2- 172.21.15.156:0/649849122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d2815c390 0x7f4d2806d660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097a0 tx=0x7f4d1802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4656e8e8ceabac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18034070 con 0x7f4d2815c390 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d18030610 con 0x7f4d2815c390 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d2d015640 1 -- 172.21.15.156:0/649849122 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d180309e0 con 0x7f4d2815c390 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 -- 172.21.15.156:0/649849122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d2815c390 msgr2=0x7f4d2806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 --2- 172.21.15.156:0/649849122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d2815c390 0x7f4d2806d660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097a0 tx=0x7f4d1802b960 comp rx=0 tx=0).stop 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 -- 172.21.15.156:0/649849122 shutdown_connections 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 --2- 172.21.15.156:0/649849122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d2815c390 0x7f4d2806d660 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 -- 172.21.15.156:0/649849122 >> 172.21.15.156:0/649849122 conn(0x7f4d280b1f30 msgr2=0x7f4d280b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 -- 172.21.15.156:0/649849122 shutdown_connections 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.486+0000 7f4d3027b640 1 -- 172.21.15.156:0/649849122 wait complete. 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.487+0000 7f4d3027b640 1 Processor -- start 2024-03-29T04:09:48.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 -- start start 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d2806dba0 0x7f4d2815a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d280d0350 0x7f4d28153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4d2815c390 0x7f4d28154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d28073700 con 0x7f4d2806dba0 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2815c810 con 0x7f4d280d0350 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d3027b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4d28159120 con 0x7f4d2815c390 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d2e017640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d2806dba0 0x7f4d2815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d2e017640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d2806dba0 0x7f4d2815a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39888/0 (socket says 172.21.15.156:39888) 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d2d816640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d280d0350 0x7f4d28153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d2e017640 1 -- 172.21.15.156:0/3020482055 learned_addr learned my addr 172.21.15.156:0/3020482055 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.488+0000 7f4d2e818640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4d2815c390 0x7f4d28154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d2d816640 1 -- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4d2815c390 msgr2=0x7f4d28154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d2d816640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4d2815c390 0x7f4d28154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d2d816640 1 -- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d2806dba0 msgr2=0x7f4d2815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d2d816640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d2806dba0 0x7f4d2815a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d2d816640 1 -- 172.21.15.156:0/3020482055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d281512c0 con 0x7f4d280d0350 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d2d816640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d280d0350 0x7f4d28153b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18034040 con 0x7f4d280d0350 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1803aaa0 con 0x7f4d280d0350 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d281546c0 con 0x7f4d280d0350 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d180443f0 con 0x7f4d280d0350 2024-03-29T04:09:48.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.489+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d28154c80 con 0x7f4d280d0350 2024-03-29T04:09:48.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.490+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d2804f9d0 con 0x7f4d280d0350 2024-03-29T04:09:48.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.491+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4d1804b060 con 0x7f4d280d0350 2024-03-29T04:09:48.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.491+0000 7f4d1effd640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4d0c06c560 0x7f4d0c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.491+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f4d180acf40 con 0x7f4d280d0350 2024-03-29T04:09:48.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.491+0000 7f4d2e017640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4d0c06c560 0x7f4d0c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.472 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.491+0000 7f4d2e017640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4d0c06c560 0x7f4d0c06ea20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d24000f60 tx=0x7f4d24007e10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.496+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4d1803f070 con 0x7f4d280d0350 2024-03-29T04:09:48.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.736+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f4d2806f4d0 con 0x7f4d280d0350 2024-03-29T04:09:48.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.741+0000 7f4d1effd640 1 -- 172.21.15.156:0/3020482055 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f4d1807b260 con 0x7f4d280d0350 2024-03-29T04:09:48.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4d0c06c560 msgr2=0x7f4d0c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4d0c06c560 0x7f4d0c06ea20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d24000f60 tx=0x7f4d24007e10 comp rx=0 tx=0).stop 2024-03-29T04:09:48.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d280d0350 msgr2=0x7f4d28153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d280d0350 0x7f4d28153b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d180046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 shutdown_connections 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4d0c06c560 0x7f4d0c06ea20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4d2815c390 0x7f4d28154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d280d0350 0x7f4d28153b10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 --2- 172.21.15.156:0/3020482055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d2806dba0 0x7f4d2815a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 >> 172.21.15.156:0/3020482055 conn(0x7f4d280b1f30 msgr2=0x7f4d280aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 shutdown_connections 2024-03-29T04:09:48.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.743+0000 7f4d3027b640 1 -- 172.21.15.156:0/3020482055 wait complete. 2024-03-29T04:09:48.733 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-29T04:09:48.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.856+0000 7fb665900640 1 Processor -- start 2024-03-29T04:09:48.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.856+0000 7fb665900640 1 -- start start 2024-03-29T04:09:48.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb665900640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6600d2120 0x7fb66006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb665900640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb66006cb10 0x7fb6600cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb665900640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6600d2df0 con 0x7fb66015e920 2024-03-29T04:09:48.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb665900640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb6600cf7a0 con 0x7fb66006cb10 2024-03-29T04:09:48.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb665900640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb6600cf190 con 0x7fb6600d2120 2024-03-29T04:09:48.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65f7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb66006cb10 0x7fb6600cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65effd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb66015e920 0x7fb6600d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44796/0 (socket says 172.21.15.156:44796) 2024-03-29T04:09:48.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65effd640 1 -- 172.21.15.156:0/1641166509 learned_addr learned my addr 172.21.15.156:0/1641166509 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:48.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65e7fc640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6600d2120 0x7fb66006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65f7fe640 1 -- 172.21.15.156:0/1641166509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6600d2120 msgr2=0x7fb66006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:48.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65f7fe640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6600d2120 0x7fb66006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65f7fe640 1 -- 172.21.15.156:0/1641166509 >> v1:172.21.15.156:6789/0 conn(0x7fb66015e920 legacy=0x7fb6600d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65dffb640 1 -- 172.21.15.156:0/1641166509 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429406175 0 0) 0x7fb6600d2df0 con 0x7fb66015e920 2024-03-29T04:09:48.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65f7fe640 1 -- 172.21.15.156:0/1641166509 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6600d3080 con 0x7fb66006cb10 2024-03-29T04:09:48.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.857+0000 7fb65e7fc640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6600d2120 0x7fb66006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:48.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.858+0000 7fb65f7fe640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66006cb10 0x7fb6600cfc30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb658009d90 tx=0x7fb65802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2771bd4c9abf895f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.858+0000 7fb65dffb640 1 -- 172.21.15.156:0/1641166509 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658039070 con 0x7fb66006cb10 2024-03-29T04:09:48.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.858+0000 7fb65dffb640 1 -- 172.21.15.156:0/1641166509 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65802cc50 con 0x7fb66006cb10 2024-03-29T04:09:48.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.858+0000 7fb65dffb640 1 -- 172.21.15.156:0/1641166509 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658034440 con 0x7fb66006cb10 2024-03-29T04:09:48.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 -- 172.21.15.156:0/1641166509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66006cb10 msgr2=0x7fb6600cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66006cb10 0x7fb6600cfc30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb658009d90 tx=0x7fb65802bea0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 -- 172.21.15.156:0/1641166509 shutdown_connections 2024-03-29T04:09:48.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66006cb10 0x7fb6600cfc30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 --2- 172.21.15.156:0/1641166509 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6600d2120 0x7fb66006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 -- 172.21.15.156:0/1641166509 >> 172.21.15.156:0/1641166509 conn(0x7fb66009d7a0 msgr2=0x7fb6600d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 -- 172.21.15.156:0/1641166509 shutdown_connections 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.859+0000 7fb665900640 1 -- 172.21.15.156:0/1641166509 wait complete. 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.860+0000 7fb665900640 1 Processor -- start 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.860+0000 7fb665900640 1 -- start start 2024-03-29T04:09:48.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb665900640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb66006cb10 0x7fb660151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb665900640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6600d2120 0x7fb6601614f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb665900640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 0x7fb66015ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb665900640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb6600d34e0 con 0x7fb66015e920 2024-03-29T04:09:48.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb665900640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6600d3360 con 0x7fb6600d2120 2024-03-29T04:09:48.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb665900640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb6600d0170 con 0x7fb66006cb10 2024-03-29T04:09:48.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 0x7fb66015ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 0x7fb66015ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39908/0 (socket says 172.21.15.156:39908) 2024-03-29T04:09:48.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 -- 172.21.15.156:0/3262898788 learned_addr learned my addr 172.21.15.156:0/3262898788 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:48.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65e7fc640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6600d2120 0x7fb6601614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65effd640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb66006cb10 0x7fb660151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 -- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb66006cb10 msgr2=0x7fb660151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb66006cb10 0x7fb660151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 -- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6600d2120 msgr2=0x7fb6601614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6600d2120 0x7fb6601614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 -- 172.21.15.156:0/3262898788 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb660161a30 con 0x7fb66015e920 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.861+0000 7fb65f7fe640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 0x7fb66015ed00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb658009eb0 tx=0x7fb658004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.862+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658039050 con 0x7fb66015e920 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.862+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb658045070 con 0x7fb66015e920 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.862+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb660152d10 con 0x7fb66015e920 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.862+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65803e610 con 0x7fb66015e920 2024-03-29T04:09:48.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.862+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66015fcb0 con 0x7fb66015e920 2024-03-29T04:09:48.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.863+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6600a2480 con 0x7fb66015e920 2024-03-29T04:09:48.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.863+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb65803e7b0 con 0x7fb66015e920 2024-03-29T04:09:48.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.864+0000 7fb647fff640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb64c06c530 0x7fb64c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:48.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.864+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fb6580adfb0 con 0x7fb66015e920 2024-03-29T04:09:48.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.864+0000 7fb65effd640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb64c06c530 0x7fb64c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:48.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.864+0000 7fb65effd640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb64c06c530 0x7fb64c06e9f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb650007fd0 tx=0x7fb650007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:48.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:48.868+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb65807c3f0 con 0x7fb66015e920 2024-03-29T04:09:49.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.113+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fb66004f9d0 con 0x7fb66015e920 2024-03-29T04:09:49.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.115+0000 7fb647fff640 1 -- 172.21.15.156:0/3262898788 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fb6580806f0 con 0x7fb66015e920 2024-03-29T04:09:49.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.116+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb64c06c530 msgr2=0x7fb64c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.116+0000 7fb665900640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb64c06c530 0x7fb64c06e9f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb650007fd0 tx=0x7fb650007de0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.116+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 msgr2=0x7fb66015ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.116+0000 7fb665900640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 0x7fb66015ed00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb658009eb0 tx=0x7fb658004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 shutdown_connections 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb64c06c530 0x7fb64c06e9f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb66015e920 0x7fb66015ed00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6600d2120 0x7fb6601614f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 --2- 172.21.15.156:0/3262898788 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb66006cb10 0x7fb660151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 >> 172.21.15.156:0/3262898788 conn(0x7fb66009d7a0 msgr2=0x7fb66015c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 shutdown_connections 2024-03-29T04:09:49.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.117+0000 7fb665900640 1 -- 172.21.15.156:0/3262898788 wait complete. 2024-03-29T04:09:49.107 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-29T04:09:49.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.230+0000 7f7311d8f640 1 Processor -- start 2024-03-29T04:09:49.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.230+0000 7f7311d8f640 1 -- start start 2024-03-29T04:09:49.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.230+0000 7f7311d8f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 0x7f730c15f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.230+0000 7f7311d8f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f730c15b8d0 con 0x7f730c15c390 2024-03-29T04:09:49.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.230+0000 7f7311d8f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f730c06e120 con 0x7f730c0d0350 2024-03-29T04:09:49.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.230+0000 7f7311d8f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f730c15bf00 con 0x7f730c15eec0 2024-03-29T04:09:49.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730affd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f730c15c390 0x7f730c06d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44798/0 (socket says 172.21.15.156:44798) 2024-03-29T04:09:49.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730affd640 1 -- 172.21.15.156:0/3026346152 learned_addr learned my addr 172.21.15.156:0/3026346152 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:49.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375766377 0 0) 0x7f730c15b8d0 con 0x7f730c15c390 2024-03-29T04:09:49.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730bfff640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 0x7f730c15f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72f4003520 con 0x7f730c15c390 2024-03-29T04:09:49.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2652408931 0 0) 0x7f72f4003520 con 0x7f730c15c390 2024-03-29T04:09:49.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 msgr2=0x7f730c15f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:49.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 0x7f730c15f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 >> v1:172.21.15.203:6789/0 conn(0x7f730c0d0350 legacy=0x7f730c0d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c06e400 con 0x7f730c15c390 2024-03-29T04:09:49.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f7300001670 con 0x7f730c15c390 2024-03-29T04:09:49.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 >> v1:172.21.15.156:6789/0 conn(0x7f730c15c390 legacy=0x7f730c06d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f40039e0 0x7f72f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f4004800 0x7f72f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 0x7f72f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f730c15bf00 con 0x7f72f40039e0 2024-03-29T04:09:49.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72f400ba70 con 0x7f72f4004800 2024-03-29T04:09:49.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f72f400de80 con 0x7f730c15eec0 2024-03-29T04:09:49.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730b7fe640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f40039e0 0x7f72f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.226 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730affd640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f4004800 0x7f72f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730bfff640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 0x7f72f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.231+0000 7f730b7fe640 1 -- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 msgr2=0x7f72f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:49.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730b7fe640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c15eec0 0x7f72f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730b7fe640 1 -- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f4004800 msgr2=0x7f72f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:49.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730b7fe640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f4004800 0x7f72f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730b7fe640 1 -- 172.21.15.156:0/3026346152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f4003520 con 0x7f72f40039e0 2024-03-29T04:09:49.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730b7fe640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f40039e0 0x7f72f4003e00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f72fc001720 tx=0x7f72f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e80473bd5c1cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc001370 con 0x7f72f40039e0 2024-03-29T04:09:49.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc0052d0 con 0x7f72f40039e0 2024-03-29T04:09:49.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.232+0000 7f730a7fc640 1 -- 172.21.15.156:0/3026346152 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc003830 con 0x7f72f40039e0 2024-03-29T04:09:49.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f7311d8f640 1 -- 172.21.15.156:0/3026346152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f40039e0 msgr2=0x7f72f4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f7311d8f640 1 --2- 172.21.15.156:0/3026346152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f40039e0 0x7f72f4003e00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f72fc001720 tx=0x7f72f4005830 comp rx=0 tx=0).stop 2024-03-29T04:09:49.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f730bfff640 1 -- 172.21.15.156:0/3026346152 reap_dead start 2024-03-29T04:09:49.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f7311d8f640 1 -- 172.21.15.156:0/3026346152 shutdown_connections 2024-03-29T04:09:49.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f7311d8f640 1 -- 172.21.15.156:0/3026346152 >> 172.21.15.156:0/3026346152 conn(0x7f730c0b1f30 msgr2=0x7f730c0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:49.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f7311d8f640 1 -- 172.21.15.156:0/3026346152 shutdown_connections 2024-03-29T04:09:49.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.233+0000 7f7311d8f640 1 -- 172.21.15.156:0/3026346152 wait complete. 2024-03-29T04:09:49.229 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.234+0000 7f7311d8f640 1 Processor -- start 2024-03-29T04:09:49.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.234+0000 7f7311d8f640 1 -- start start 2024-03-29T04:09:49.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f7311d8f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f40039e0 0x7f730c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f7311d8f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c153ff0 0x7f730c0b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f7311d8f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 0x7f730c159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f7311d8f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f730c073700 con 0x7f730c0b13d0 2024-03-29T04:09:49.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f7311d8f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f730c15c810 con 0x7f72f40039e0 2024-03-29T04:09:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f7311d8f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f730c06dd70 con 0x7f730c153ff0 2024-03-29T04:09:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 0x7f730c159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 0x7f730c159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39940/0 (socket says 172.21.15.156:39940) 2024-03-29T04:09:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 -- 172.21.15.156:0/2123524251 learned_addr learned my addr 172.21.15.156:0/2123524251 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730affd640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c153ff0 0x7f730c0b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730b7fe640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f40039e0 0x7f730c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 -- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c153ff0 msgr2=0x7f730c0b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c153ff0 0x7f730c0b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 -- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f40039e0 msgr2=0x7f730c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f40039e0 0x7f730c14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.235+0000 7f730bfff640 1 -- 172.21.15.156:0/2123524251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c154aa0 con 0x7f730c0b13d0 2024-03-29T04:09:49.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.236+0000 7f730bfff640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 0x7f730c159270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f730c0b30e0 tx=0x7f730400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.236+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304015430 con 0x7f730c0b13d0 2024-03-29T04:09:49.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.236+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73040155f0 con 0x7f730c0b13d0 2024-03-29T04:09:49.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.236+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73040159e0 con 0x7f730c0b13d0 2024-03-29T04:09:49.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.236+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c154d80 con 0x7f730c0b13d0 2024-03-29T04:09:49.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.236+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c0b0970 con 0x7f730c0b13d0 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.237+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c0b0b70 con 0x7f730c0b13d0 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.237+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7304015b80 con 0x7f730c0b13d0 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.237+0000 7f72f3fff640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f72f806c530 0x7f72f806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.238+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f730408c020 con 0x7f730c0b13d0 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.238+0000 7f730b7fe640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f72f806c530 0x7f72f806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.238+0000 7f730b7fe640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f72f806c530 0x7f72f806e9f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f72fc033000 tx=0x7f72fc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.235 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.242+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f730c0b0b70 con 0x7f730c0b13d0 2024-03-29T04:09:49.464 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.484+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f730c072510 con 0x7f730c0b13d0 2024-03-29T04:09:49.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.486+0000 7f72f3fff640 1 -- 172.21.15.156:0/2123524251 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f7304019030 con 0x7f730c0b13d0 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f72f806c530 msgr2=0x7f72f806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f72f806c530 0x7f72f806e9f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f72fc033000 tx=0x7f72fc009030 comp rx=0 tx=0).stop 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 msgr2=0x7f730c159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 0x7f730c159270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f730c0b30e0 tx=0x7f730400f580 comp rx=0 tx=0).stop 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 shutdown_connections 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f72f806c530 0x7f72f806e9f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f730c0b13d0 0x7f730c159270 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f730c153ff0 0x7f730c0b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 --2- 172.21.15.156:0/2123524251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72f40039e0 0x7f730c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 >> 172.21.15.156:0/2123524251 conn(0x7f730c0b1f30 msgr2=0x7f730c15c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 shutdown_connections 2024-03-29T04:09:49.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.488+0000 7f7311d8f640 1 -- 172.21.15.156:0/2123524251 wait complete. 2024-03-29T04:09:49.478 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-29T04:09:49.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.601+0000 7fc895104640 1 Processor -- start 2024-03-29T04:09:49.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.601+0000 7fc895104640 1 -- start start 2024-03-29T04:09:49.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc895104640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc890142750 0x7fc89015d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc895104640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc89006ccd0 0x7fc8900cd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc895104640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc89015c980 con 0x7fc8900cd460 2024-03-29T04:09:49.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc895104640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc890142b30 con 0x7fc890142750 2024-03-29T04:09:49.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc895104640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc89015d960 con 0x7fc89006ccd0 2024-03-29T04:09:49.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc890142750 0x7fc89015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc890142750 0x7fc89015d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39950/0 (socket says 172.21.15.156:39950) 2024-03-29T04:09:49.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 -- 172.21.15.156:0/1983940214 learned_addr learned my addr 172.21.15.156:0/1983940214 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:49.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88f59e640 1 --2- 172.21.15.156:0/1983940214 >> v2:172.21.15.203:3300/0 conn(0x7fc89006ccd0 0x7fc8900cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 -- 172.21.15.156:0/1983940214 >> v2:172.21.15.203:3300/0 conn(0x7fc89006ccd0 msgr2=0x7fc8900cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:49.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 --2- 172.21.15.156:0/1983940214 >> v2:172.21.15.203:3300/0 conn(0x7fc89006ccd0 0x7fc8900cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 -- 172.21.15.156:0/1983940214 >> v1:172.21.15.203:6789/0 conn(0x7fc8900cd460 legacy=0x7fc89006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:49.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.602+0000 7fc88ed9d640 1 -- 172.21.15.156:0/1983940214 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8900cdf10 con 0x7fc890142750 2024-03-29T04:09:49.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.603+0000 7fc88ed9d640 1 --2- 172.21.15.156:0/1983940214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89015d420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc8900677b0 tx=0x7fc88002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab132e8c44703eb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.603+0000 7fc88dd9b640 1 -- 172.21.15.156:0/1983940214 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88002c9f0 con 0x7fc890142750 2024-03-29T04:09:49.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.603+0000 7fc88dd9b640 1 -- 172.21.15.156:0/1983940214 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88002cb90 con 0x7fc890142750 2024-03-29T04:09:49.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.603+0000 7fc88dd9b640 1 -- 172.21.15.156:0/1983940214 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8800343f0 con 0x7fc890142750 2024-03-29T04:09:49.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 -- 172.21.15.156:0/1983940214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 msgr2=0x7fc89015d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 --2- 172.21.15.156:0/1983940214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89015d420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc8900677b0 tx=0x7fc88002be50 comp rx=0 tx=0).stop 2024-03-29T04:09:49.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 -- 172.21.15.156:0/1983940214 shutdown_connections 2024-03-29T04:09:49.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 --2- 172.21.15.156:0/1983940214 >> v2:172.21.15.203:3300/0 conn(0x7fc89006ccd0 0x7fc8900cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 --2- 172.21.15.156:0/1983940214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89015d420 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 -- 172.21.15.156:0/1983940214 >> 172.21.15.156:0/1983940214 conn(0x7fc890072110 msgr2=0x7fc8900724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 -- 172.21.15.156:0/1983940214 shutdown_connections 2024-03-29T04:09:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.604+0000 7fc895104640 1 -- 172.21.15.156:0/1983940214 wait complete. 2024-03-29T04:09:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.605+0000 7fc895104640 1 Processor -- start 2024-03-29T04:09:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.605+0000 7fc895104640 1 -- start start 2024-03-29T04:09:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc895104640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc89006ccd0 0x7fc8900b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc895104640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8900cd460 0x7fc89006e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc895104640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89006e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc895104640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc89015cf00 con 0x7fc890142750 2024-03-29T04:09:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc895104640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc89015cc20 con 0x7fc89006ccd0 2024-03-29T04:09:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc895104640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc8900cce90 con 0x7fc8900cd460 2024-03-29T04:09:49.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc88f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89006e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc88f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89006e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39962/0 (socket says 172.21.15.156:39962) 2024-03-29T04:09:49.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc88f59e640 1 -- 172.21.15.156:0/2799233404 learned_addr learned my addr 172.21.15.156:0/2799233404 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:49.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc88ed9d640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc89006ccd0 0x7fc8900b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.606+0000 7fc88e59c640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8900cd460 0x7fc89006e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88f59e640 1 -- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8900cd460 msgr2=0x7fc89006e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88f59e640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8900cd460 0x7fc89006e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88f59e640 1 -- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc89006ccd0 msgr2=0x7fc8900b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88f59e640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc89006ccd0 0x7fc8900b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88f59e640 1 -- 172.21.15.156:0/2799233404 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89006ec40 con 0x7fc890142750 2024-03-29T04:09:49.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88e59c640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8900cd460 0x7fc89006e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:49.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc88f59e640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89006e610 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc88800d800 tx=0x7fc88800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88800b6c0 con 0x7fc890142750 2024-03-29T04:09:49.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc888019070 con 0x7fc890142750 2024-03-29T04:09:49.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88800bac0 con 0x7fc890142750 2024-03-29T04:09:49.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89006ef20 con 0x7fc890142750 2024-03-29T04:09:49.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.607+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc89006f7c0 con 0x7fc890142750 2024-03-29T04:09:49.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.608+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc89006f9c0 con 0x7fc890142750 2024-03-29T04:09:49.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.608+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc88800bc60 con 0x7fc890142750 2024-03-29T04:09:49.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.609+0000 7fc8777fe640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc87c06c530 0x7fc87c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.609+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fc88808b740 con 0x7fc890142750 2024-03-29T04:09:49.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.609+0000 7fc88ed9d640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc87c06c530 0x7fc87c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.609+0000 7fc88ed9d640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc87c06c530 0x7fc87c06e9f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc880035000 tx=0x7fc880030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.613+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc89006f9c0 con 0x7fc890142750 2024-03-29T04:09:49.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.859+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fc890073740 con 0x7fc890142750 2024-03-29T04:09:49.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.861+0000 7fc8777fe640 1 -- 172.21.15.156:0/2799233404 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fc88805a0b0 con 0x7fc890142750 2024-03-29T04:09:49.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc87c06c530 msgr2=0x7fc87c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc87c06c530 0x7fc87c06e9f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc880035000 tx=0x7fc880030000 comp rx=0 tx=0).stop 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 msgr2=0x7fc89006e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89006e610 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc88800d800 tx=0x7fc88800dcc0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 shutdown_connections 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc87c06c530 0x7fc87c06e9f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc890142750 0x7fc89006e610 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8900cd460 0x7fc89006e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 --2- 172.21.15.156:0/2799233404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc89006ccd0 0x7fc8900b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 >> 172.21.15.156:0/2799233404 conn(0x7fc890072110 msgr2=0x7fc89015a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:49.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 shutdown_connections 2024-03-29T04:09:49.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.863+0000 7fc895104640 1 -- 172.21.15.156:0/2799233404 wait complete. 2024-03-29T04:09:49.854 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-29T04:09:49.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.976+0000 7fc243888640 1 Processor -- start 2024-03-29T04:09:49.955 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.976+0000 7fc243888640 1 -- start start 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc243888640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc23c0af6b0 0x7fc23c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc243888640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc23c15c7e0 con 0x7fc23c0b3050 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc243888640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc23c0afa90 con 0x7fc23c06f7a0 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc243888640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc23c0b3510 con 0x7fc23c0af6b0 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc241624640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc23c0af6b0 0x7fc23c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc240e23640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc23c0b3050 0x7fc23c06f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44800/0 (socket says 172.21.15.156:44800) 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc241624640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc23c0af6b0 0x7fc23c15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39964/0 (socket says 172.21.15.156:39964) 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc240e23640 1 -- 172.21.15.156:0/2581115393 learned_addr learned my addr 172.21.15.156:0/2581115393 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc241624640 1 -- 172.21.15.156:0/2581115393 >> v1:172.21.15.203:6789/0 conn(0x7fc23c06f7a0 legacy=0x7fc23c071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc241624640 1 -- 172.21.15.156:0/2581115393 >> v1:172.21.15.156:6789/0 conn(0x7fc23c0b3050 legacy=0x7fc23c06f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc241624640 1 -- 172.21.15.156:0/2581115393 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c0b37f0 con 0x7fc23c0af6b0 2024-03-29T04:09:49.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.977+0000 7fc233fff640 1 -- 172.21.15.156:0/2581115393 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881913015 0 0) 0x7fc22c001110 con 0x7fc23c0b3050 2024-03-29T04:09:49.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.978+0000 7fc241624640 1 --2- 172.21.15.156:0/2581115393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c0af6b0 0x7fc23c15c2a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc2380097a0 tx=0x7fc23802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcdf65ca29d4bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.978+0000 7fc233fff640 1 -- 172.21.15.156:0/2581115393 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238035020 con 0x7fc23c0af6b0 2024-03-29T04:09:49.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.978+0000 7fc233fff640 1 -- 172.21.15.156:0/2581115393 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc238031830 con 0x7fc23c0af6b0 2024-03-29T04:09:49.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.978+0000 7fc233fff640 1 -- 172.21.15.156:0/2581115393 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238031c00 con 0x7fc23c0af6b0 2024-03-29T04:09:49.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.978+0000 7fc243888640 1 -- 172.21.15.156:0/2581115393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c0af6b0 msgr2=0x7fc23c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.978+0000 7fc243888640 1 --2- 172.21.15.156:0/2581115393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c0af6b0 0x7fc23c15c2a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc2380097a0 tx=0x7fc23802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.979+0000 7fc243888640 1 -- 172.21.15.156:0/2581115393 shutdown_connections 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.979+0000 7fc243888640 1 --2- 172.21.15.156:0/2581115393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c0af6b0 0x7fc23c15c2a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.979+0000 7fc243888640 1 -- 172.21.15.156:0/2581115393 >> 172.21.15.156:0/2581115393 conn(0x7fc23c0b1d50 msgr2=0x7fc23c0b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.979+0000 7fc243888640 1 -- 172.21.15.156:0/2581115393 shutdown_connections 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.979+0000 7fc243888640 1 -- 172.21.15.156:0/2581115393 wait complete. 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 Processor -- start 2024-03-29T04:09:49.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 -- start start 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 0x7fc23c09dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0af6b0 0x7fc23c159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c0b3050 0x7fc23c06df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c0b4090 con 0x7fc23c06f7a0 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c0b3db0 con 0x7fc23c0af6b0 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.980+0000 7fc243888640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc23c0b45b0 con 0x7fc23c0b3050 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 0x7fc23c09dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 0x7fc23c09dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39972/0 (socket says 172.21.15.156:39972) 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 -- 172.21.15.156:0/3900356401 learned_addr learned my addr 172.21.15.156:0/3900356401 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 -- 172.21.15.156:0/3900356401 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c0b3050 msgr2=0x7fc23c06df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c0b3050 0x7fc23c06df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 -- 172.21.15.156:0/3900356401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0af6b0 msgr2=0x7fc23c159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0af6b0 0x7fc23c159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 -- 172.21.15.156:0/3900356401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c148d20 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc241624640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 0x7fc23c09dbf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc2380098c0 tx=0x7fc2380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238041070 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23803aa30 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238035040 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23c15a1f0 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.981+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc23c09e130 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.982+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23c068610 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.983+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc23804c060 con 0x7fc23c06f7a0 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.983+0000 7fc231ffb640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22006c530 0x7fc22006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:49.964 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.983+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fc2380ace40 con 0x7fc23c06f7a0 2024-03-29T04:09:49.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.983+0000 7fc240e23640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22006c530 0x7fc22006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:49.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.984+0000 7fc240e23640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22006c530 0x7fc22006e9f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc22c000c00 tx=0x7fc22c0079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:49.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:49.988+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc23807b7b0 con 0x7fc23c06f7a0 2024-03-29T04:09:50.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.228+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fc23c149040 con 0x7fc23c06f7a0 2024-03-29T04:09:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.230+0000 7fc231ffb640 1 -- 172.21.15.156:0/3900356401 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fc23807b0b0 con 0x7fc23c06f7a0 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22006c530 msgr2=0x7fc22006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22006c530 0x7fc22006e9f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc22c000c00 tx=0x7fc22c0079c0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 msgr2=0x7fc23c09dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 0x7fc23c09dbf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc2380098c0 tx=0x7fc2380046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 shutdown_connections 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22006c530 0x7fc22006e9f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c0b3050 0x7fc23c06df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0af6b0 0x7fc23c159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 --2- 172.21.15.156:0/3900356401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23c06f7a0 0x7fc23c09dbf0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 >> 172.21.15.156:0/3900356401 conn(0x7fc23c0b1d50 msgr2=0x7fc23c0b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:50.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 shutdown_connections 2024-03-29T04:09:50.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.232+0000 7fc243888640 1 -- 172.21.15.156:0/3900356401 wait complete. 2024-03-29T04:09:50.222 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-29T04:09:50.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.344+0000 7f055f8e0640 1 Processor -- start 2024-03-29T04:09:50.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.345+0000 7f055f8e0640 1 -- start start 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.345+0000 7f055f8e0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05580d2120 0x7f055806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.345+0000 7f055f8e0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05580d2df0 con 0x7f055806cb10 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.345+0000 7f055f8e0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05580cf190 con 0x7f055815e920 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.345+0000 7f055f8e0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f05580cf7a0 con 0x7f05580d2120 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055de7d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f055806cb10 0x7f05580cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44808/0 (socket says 172.21.15.156:44808) 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055de7d640 1 -- 172.21.15.156:0/212265722 learned_addr learned my addr 172.21.15.156:0/212265722 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055ce7b640 1 --2- 172.21.15.156:0/212265722 >> v2:172.21.15.203:3300/0 conn(0x7f05580d2120 0x7f055806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823774710 0 0) 0x7f05580d2df0 con 0x7f055806cb10 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0544003520 con 0x7f055806cb10 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2198511000 0 0) 0x7f0544003520 con 0x7f055806cb10 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 >> v2:172.21.15.203:3300/0 conn(0x7f05580d2120 msgr2=0x7f055806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 --2- 172.21.15.156:0/212265722 >> v2:172.21.15.203:3300/0 conn(0x7f05580d2120 0x7f055806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.328 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 >> v1:172.21.15.203:6789/0 conn(0x7f055815e920 legacy=0x7f05580d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05580d3080 con 0x7f055806cb10 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f05500019f0 con 0x7f055806cb10 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 >> v1:172.21.15.156:6789/0 conn(0x7f055806cb10 legacy=0x7f05580cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f0544003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0544004800 0x7f0544004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05440051c0 0x7f054400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05580cf7a0 con 0x7f05440039b0 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f054400c060 con 0x7f05440051c0 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f054400c260 con 0x7f0544004800 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055d67c640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f0544003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055de7d640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05440051c0 0x7f054400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055ce7b640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0544004800 0x7f0544004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055d67c640 1 -- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0544004800 msgr2=0x7f0544004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055d67c640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0544004800 0x7f0544004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.329 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055d67c640 1 -- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05440051c0 msgr2=0x7f054400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055d67c640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05440051c0 0x7f054400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055d67c640 1 -- 172.21.15.156:0/212265722 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0544003520 con 0x7f05440039b0 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055de7d640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05440051c0 0x7f054400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.346+0000 7f055de7d640 1 -- 172.21.15.156:0/212265722 reap_dead start 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.347+0000 7f055d67c640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f0544003dd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0548001750 tx=0x7f0544005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9d7d64fab2b957 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.347+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548001370 con 0x7f05440039b0 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.347+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05480052d0 con 0x7f05440039b0 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.347+0000 7f054ffff640 1 -- 172.21.15.156:0/212265722 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548003830 con 0x7f05440039b0 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 -- 172.21.15.156:0/212265722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 msgr2=0x7f0544003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f0544003dd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0548001750 tx=0x7f0544005830 comp rx=0 tx=0).stop 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 -- 172.21.15.156:0/212265722 shutdown_connections 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 --2- 172.21.15.156:0/212265722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f0544003dd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 -- 172.21.15.156:0/212265722 >> 172.21.15.156:0/212265722 conn(0x7f055809d7a0 msgr2=0x7f05580d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 -- 172.21.15.156:0/212265722 shutdown_connections 2024-03-29T04:09:50.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.348+0000 7f055f8e0640 1 -- 172.21.15.156:0/212265722 wait complete. 2024-03-29T04:09:50.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.349+0000 7f055f8e0640 1 Processor -- start 2024-03-29T04:09:50.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.349+0000 7f055f8e0640 1 -- start start 2024-03-29T04:09:50.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055f8e0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f05580cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055f8e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f05580cdce0 0x7f05580cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055f8e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05580cce10 0x7f055814b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055f8e0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05580d34e0 con 0x7f05440039b0 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055f8e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05580d3360 con 0x7f05580cce10 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055f8e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f055806c5d0 con 0x7f05580cdce0 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f05580cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f05580cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39982/0 (socket says 172.21.15.156:39982) 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 -- 172.21.15.156:0/3188465055 learned_addr learned my addr 172.21.15.156:0/3188465055 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 -- 172.21.15.156:0/3188465055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f05580cdce0 msgr2=0x7f05580cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f05580cdce0 0x7f05580cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 -- 172.21.15.156:0/3188465055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05580cce10 msgr2=0x7f055814b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05580cce10 0x7f055814b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 -- 172.21.15.156:0/3188465055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05581606b0 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f055d67c640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f05580cd7a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0548005040 tx=0x7f0548001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.350+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548041070 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.351+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0548005480 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.351+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0558149a20 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.351+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548007a30 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.351+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0558149fe0 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.352+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05580a2480 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.352+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0548044070 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.352+0000 7f054dffb640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f053c06c560 0x7f053c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.353+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f05480acd10 con 0x7f05440039b0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.353+0000 7f055ce7b640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f053c06c560 0x7f053c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.353+0000 7f055ce7b640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f053c06c560 0x7f053c06ea20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0558161d90 tx=0x7f0554008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:50.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.357+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f05581606b0 con 0x7f05440039b0 2024-03-29T04:09:50.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.601+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f055804f9d0 con 0x7f05440039b0 2024-03-29T04:09:50.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.603+0000 7f054dffb640 1 -- 172.21.15.156:0/3188465055 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f054807b830 con 0x7f05440039b0 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f053c06c560 msgr2=0x7f053c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f053c06c560 0x7f053c06ea20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0558161d90 tx=0x7f0554008040 comp rx=0 tx=0).stop 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 msgr2=0x7f05580cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f05580cd7a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0548005040 tx=0x7f0548001120 comp rx=0 tx=0).stop 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 shutdown_connections 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f053c06c560 0x7f053c06ea20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05580cce10 0x7f055814b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f05580cdce0 0x7f05580cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 --2- 172.21.15.156:0/3188465055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05440039b0 0x7f05580cd7a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 >> 172.21.15.156:0/3188465055 conn(0x7f055809d7a0 msgr2=0x7f05580d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:50.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 shutdown_connections 2024-03-29T04:09:50.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.605+0000 7f055f8e0640 1 -- 172.21.15.156:0/3188465055 wait complete. 2024-03-29T04:09:50.595 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:50.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.716+0000 7f77cc1a8640 1 Processor -- start 2024-03-29T04:09:50.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.716+0000 7f77cc1a8640 1 -- start start 2024-03-29T04:09:50.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77cc1a8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f77c4142690 0x7f77c40ad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77cc1a8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f77c40ad570 0x7f77c40b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77cc1a8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77c40b30f0 con 0x7f77c415c270 2024-03-29T04:09:50.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77cc1a8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f77c40ad9a0 con 0x7f77c4142690 2024-03-29T04:09:50.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77cc1a8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f77c415cd60 con 0x7f77c40ad570 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9743640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f77c4142690 0x7f77c40ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9f44640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f77c415c270 0x7f77c415c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44824/0 (socket says 172.21.15.156:44824) 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9f44640 1 -- 172.21.15.156:0/3684074903 learned_addr learned my addr 172.21.15.156:0/3684074903 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77ca745640 1 --2- 172.21.15.156:0/3684074903 >> v2:172.21.15.203:3300/0 conn(0x7f77c40ad570 0x7f77c40b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9743640 1 -- 172.21.15.156:0/3684074903 >> v2:172.21.15.203:3300/0 conn(0x7f77c40ad570 msgr2=0x7f77c40b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9743640 1 --2- 172.21.15.156:0/3684074903 >> v2:172.21.15.203:3300/0 conn(0x7f77c40ad570 0x7f77c40b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9743640 1 -- 172.21.15.156:0/3684074903 >> v1:172.21.15.156:6789/0 conn(0x7f77c415c270 legacy=0x7f77c415c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c8f42640 1 -- 172.21.15.156:0/3684074903 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313522118 0 0) 0x7f77c40b30f0 con 0x7f77c415c270 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.717+0000 7f77c9743640 1 -- 172.21.15.156:0/3684074903 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c415d020 con 0x7f77c4142690 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.718+0000 7f77c9743640 1 --2- 172.21.15.156:0/3684074903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c40ad030 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f77c406d040 tx=0x7f77b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1163177abba90e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.718+0000 7f77c8f42640 1 -- 172.21.15.156:0/3684074903 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b40315a0 con 0x7f77c4142690 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.718+0000 7f77c8f42640 1 -- 172.21.15.156:0/3684074903 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b4037070 con 0x7f77c4142690 2024-03-29T04:09:50.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.718+0000 7f77c8f42640 1 -- 172.21.15.156:0/3684074903 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4031960 con 0x7f77c4142690 2024-03-29T04:09:50.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 -- 172.21.15.156:0/3684074903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 msgr2=0x7f77c40ad030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/3684074903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c40ad030 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f77c406d040 tx=0x7f77b402b840 comp rx=0 tx=0).stop 2024-03-29T04:09:50.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 -- 172.21.15.156:0/3684074903 shutdown_connections 2024-03-29T04:09:50.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/3684074903 >> v2:172.21.15.203:3300/0 conn(0x7f77c40ad570 0x7f77c40b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/3684074903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c40ad030 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 -- 172.21.15.156:0/3684074903 >> 172.21.15.156:0/3684074903 conn(0x7f77c40b1d50 msgr2=0x7f77c40b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 -- 172.21.15.156:0/3684074903 shutdown_connections 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.719+0000 7f77cc1a8640 1 -- 172.21.15.156:0/3684074903 wait complete. 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.720+0000 7f77cc1a8640 1 Processor -- start 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.720+0000 7f77cc1a8640 1 -- start start 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.720+0000 7f77cc1a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c40ad570 0x7f77c40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77c415c270 0x7f77c4144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4142b10 con 0x7f77c4142690 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77c415d5e0 con 0x7f77c40ad570 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f77c415d8c0 con 0x7f77c415c270 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c415e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40010/0 (socket says 172.21.15.156:40010) 2024-03-29T04:09:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 -- 172.21.15.156:0/1680864516 learned_addr learned my addr 172.21.15.156:0/1680864516 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9f44640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c40ad570 0x7f77c40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77ca745640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77c415c270 0x7f77c4144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 -- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77c415c270 msgr2=0x7f77c4144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77c415c270 0x7f77c4144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 -- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c40ad570 msgr2=0x7f77c40c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c40ad570 0x7f77c40c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 -- 172.21.15.156:0/1680864516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c41f4f80 con 0x7f77c4142690 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77ca745640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77c415c270 0x7f77c4144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77c9743640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c415e570 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f77b4000c00 tx=0x7f77b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4032840 con 0x7f77c4142690 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b4032a00 con 0x7f77c4142690 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4032df0 con 0x7f77c4142690 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c41f52c0 con 0x7f77c4142690 2024-03-29T04:09:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.721+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c4158110 con 0x7f77c4142690 2024-03-29T04:09:50.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.722+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c4068610 con 0x7f77c4142690 2024-03-29T04:09:50.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.723+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f77b404d3f0 con 0x7f77c4142690 2024-03-29T04:09:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.723+0000 7f77ba7fc640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f77a806c530 0x7f77a806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.723+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f77b402f030 con 0x7f77c4142690 2024-03-29T04:09:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.723+0000 7f77c9f44640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f77a806c530 0x7f77a806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.724+0000 7f77c9f44640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f77a806c530 0x7f77a806e9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f77c0000f60 tx=0x7f77c0007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:50.707 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.728+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f77c41f52c0 con 0x7f77c4142690 2024-03-29T04:09:50.881 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.902+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77c40b4d60 con 0x7f77c4142690 2024-03-29T04:09:50.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.903+0000 7f77ba7fc640 1 -- 172.21.15.156:0/1680864516 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+12915 (secure 0 0 0) 0x7f77b4040070 con 0x7f77c4142690 2024-03-29T04:09:50.882 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:50.882 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":30,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:47.999626+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f77a806c530 msgr2=0x7f77a806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f77a806c530 0x7f77a806e9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f77c0000f60 tx=0x7f77c0007c00 comp rx=0 tx=0).stop 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 msgr2=0x7f77c415e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c415e570 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f77b4000c00 tx=0x7f77b40046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 shutdown_connections 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f77a806c530 0x7f77a806e9f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77c415c270 0x7f77c4144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77c4142690 0x7f77c415e570 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 --2- 172.21.15.156:0/1680864516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c40ad570 0x7f77c40c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 >> 172.21.15.156:0/1680864516 conn(0x7f77c40b1d50 msgr2=0x7f77c40b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:50.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 shutdown_connections 2024-03-29T04:09:50.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:50.905+0000 7f77cc1a8640 1 -- 172.21.15.156:0/1680864516 wait complete. 2024-03-29T04:09:50.896 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:09:50.896 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:09:51.000 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.021+0000 7f4bf7814640 1 Processor -- start 2024-03-29T04:09:51.000 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.021+0000 7f4bf7814640 1 -- start start 2024-03-29T04:09:51.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf7814640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4bf015e540 0x7f4bf00b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf7814640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf00b4020 con 0x7f4bf006c5c0 2024-03-29T04:09:51.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf7814640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf00b42c0 con 0x7f4bf00b59d0 2024-03-29T04:09:51.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf7814640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4bf0070680 con 0x7f4bf015e540 2024-03-29T04:09:51.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf5db1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4bf006c5c0 0x7f4bf006e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44836/0 (socket says 172.21.15.156:44836) 2024-03-29T04:09:51.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf5db1640 1 -- 172.21.15.156:0/2982914780 learned_addr learned my addr 172.21.15.156:0/2982914780 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4292125618 0 0) 0x7f4bf00b4020 con 0x7f4bf006c5c0 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bdc003520 con 0x7f4bf006c5c0 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.022+0000 7f4bf55b0640 1 --2- 172.21.15.156:0/2982914780 >> v2:172.21.15.203:3300/0 conn(0x7f4bf015e540 0x7f4bf00b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3532419327 0 0) 0x7f4bdc003520 con 0x7f4bf006c5c0 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 >> v2:172.21.15.203:3300/0 conn(0x7f4bf015e540 msgr2=0x7f4bf00b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 --2- 172.21.15.156:0/2982914780 >> v2:172.21.15.203:3300/0 conn(0x7f4bf015e540 0x7f4bf00b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 >> v1:172.21.15.203:6789/0 conn(0x7f4bf00b59d0 legacy=0x7f4bf00ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf00b45a0 con 0x7f4bf006c5c0 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f4be8001690 con 0x7f4bf006c5c0 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 >> v1:172.21.15.156:6789/0 conn(0x7f4bf006c5c0 legacy=0x7f4bf006e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0039b0 0x7f4bdc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bdc004800 0x7f4bdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bdc0051c0 0x7f4bdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf0070680 con 0x7f4bdc0051c0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc00c0f0 con 0x7f4bdc0039b0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4bdc00c2f0 con 0x7f4bdc004800 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bdc0051c0 0x7f4bdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf55b0640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0039b0 0x7f4bdc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf4daf640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bdc004800 0x7f4bdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf5db1640 1 -- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bdc004800 msgr2=0x7f4bdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bdc004800 0x7f4bdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf5db1640 1 -- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0039b0 msgr2=0x7f4bdc003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0039b0 0x7f4bdc003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.023+0000 7f4bf5db1640 1 -- 172.21.15.156:0/2982914780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc003520 con 0x7f4bdc0051c0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4bf5db1640 1 -- 172.21.15.156:0/2982914780 reap_dead start 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bdc0051c0 0x7f4bdc00b9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00b41f0 tx=0x7f4bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93410158d4b680f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8007d30 con 0x7f4bdc0051c0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be8030070 con 0x7f4bdc0051c0 2024-03-29T04:09:51.006 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4be7fff640 1 -- 172.21.15.156:0/2982914780 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8005d10 con 0x7f4bdc0051c0 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4bf7814640 1 -- 172.21.15.156:0/2982914780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bdc0051c0 msgr2=0x7f4bdc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4bf7814640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bdc0051c0 0x7f4bdc00b9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00b41f0 tx=0x7f4bdc005830 comp rx=0 tx=0).stop 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4bf7814640 1 -- 172.21.15.156:0/2982914780 shutdown_connections 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.024+0000 7f4bf7814640 1 --2- 172.21.15.156:0/2982914780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bdc0051c0 0x7f4bdc00b9c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.025+0000 7f4bf7814640 1 -- 172.21.15.156:0/2982914780 >> 172.21.15.156:0/2982914780 conn(0x7f4bf0072110 msgr2=0x7f4bf00724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.025+0000 7f4bf7814640 1 -- 172.21.15.156:0/2982914780 shutdown_connections 2024-03-29T04:09:51.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.025+0000 7f4bf7814640 1 -- 172.21.15.156:0/2982914780 wait complete. 2024-03-29T04:09:51.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 Processor -- start 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 -- start start 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0051c0 0x7f4bf01477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bf00cee00 0x7f4bf00cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 0x7f4bf00b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf015e920 con 0x7f4bf00afc00 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf006e250 con 0x7f4bdc0051c0 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.026+0000 7f4bf7814640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf015eaa0 con 0x7f4bf00cee00 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 0x7f4bf00b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 0x7f4bf00b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40018/0 (socket says 172.21.15.156:40018) 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 -- 172.21.15.156:0/382739165 learned_addr learned my addr 172.21.15.156:0/382739165 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf55b0640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0051c0 0x7f4bf01477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 -- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bf00cee00 msgr2=0x7f4bf00cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bf00cee00 0x7f4bf00cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 -- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0051c0 msgr2=0x7f4bf01477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0051c0 0x7f4bf01477a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 -- 172.21.15.156:0/382739165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf0147ce0 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf55b0640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0051c0 0x7f4bf01477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf5db1640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 0x7f4bf00b00b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4be8001990 tx=0x7f4bf00b4770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be802de20 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be80049f0 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf00b09a0 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8004da0 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.027+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf0148370 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.029+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf0073740 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.029+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4be803e030 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.029+0000 7f4be5ffb640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4bd406c560 0x7f4bd406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.029+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f4be80abca0 con 0x7f4bf00afc00 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.029+0000 7f4bf55b0640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4bd406c560 0x7f4bd406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.030+0000 7f4bf55b0640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4bd406c560 0x7f4bd406ea20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4bec003aa0 tx=0x7f4bec009760 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.034+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4be807a610 con 0x7f4bf00afc00 2024-03-29T04:09:51.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.210+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4bf004f9d0 con 0x7f4bf00afc00 2024-03-29T04:09:51.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.211+0000 7f4be5ffb640 1 -- 172.21.15.156:0/382739165 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f4be8079f10 con 0x7f4bf00afc00 2024-03-29T04:09:51.190 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4bd406c560 msgr2=0x7f4bd406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4bd406c560 0x7f4bd406ea20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4bec003aa0 tx=0x7f4bec009760 comp rx=0 tx=0).stop 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 msgr2=0x7f4bf00b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 0x7f4bf00b00b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4be8001990 tx=0x7f4bf00b4770 comp rx=0 tx=0).stop 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 shutdown_connections 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4bd406c560 0x7f4bd406ea20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf00afc00 0x7f4bf00b00b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4bf00cee00 0x7f4bf00cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 --2- 172.21.15.156:0/382739165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bdc0051c0 0x7f4bf01477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 >> 172.21.15.156:0/382739165 conn(0x7f4bf0072110 msgr2=0x7f4bf006f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:51.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 shutdown_connections 2024-03-29T04:09:51.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.213+0000 7f4bf7814640 1 -- 172.21.15.156:0/382739165 wait complete. 2024-03-29T04:09:51.203 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T04:09:51.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.324+0000 7fe94f5d4640 1 Processor -- start 2024-03-29T04:09:51.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.324+0000 7fe94f5d4640 1 -- start start 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94f5d4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe948161ac0 0x7fe9480cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94f5d4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe9480cf5a0 0x7fe9480d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94f5d4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9480cfa60 con 0x7fe948161ea0 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94f5d4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe9480d3550 con 0x7fe948161ac0 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94f5d4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe9480cfd40 con 0x7fe9480cf5a0 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94d370640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe948161ac0 0x7fe9480cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94db71640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fe948161ea0 0x7fe948162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44846/0 (socket says 172.21.15.156:44846) 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.325+0000 7fe94db71640 1 -- 172.21.15.156:0/429272635 learned_addr learned my addr 172.21.15.156:0/429272635 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe94d370640 1 -- 172.21.15.156:0/429272635 >> v2:172.21.15.203:3300/0 conn(0x7fe9480cf5a0 msgr2=0x7fe9480d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe94cb6f640 1 --2- 172.21.15.156:0/429272635 >> v2:172.21.15.203:3300/0 conn(0x7fe9480cf5a0 0x7fe9480d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe94d370640 1 --2- 172.21.15.156:0/429272635 >> v2:172.21.15.203:3300/0 conn(0x7fe9480cf5a0 0x7fe9480d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe94d370640 1 -- 172.21.15.156:0/429272635 >> v1:172.21.15.156:6789/0 conn(0x7fe948161ea0 legacy=0x7fe948162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe93ffff640 1 -- 172.21.15.156:0/429272635 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033837618 0 0) 0x7fe9480cfa60 con 0x7fe948161ea0 2024-03-29T04:09:51.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe94d370640 1 -- 172.21.15.156:0/429272635 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9480d0020 con 0x7fe948161ac0 2024-03-29T04:09:51.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.326+0000 7fe94d370640 1 --2- 172.21.15.156:0/429272635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ac0 0x7fe9480cf060 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe94808ce30 tx=0x7fe94402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8106dd4fd2d58f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe93ffff640 1 -- 172.21.15.156:0/429272635 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944039070 con 0x7fe948161ac0 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe93ffff640 1 -- 172.21.15.156:0/429272635 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe94402cc80 con 0x7fe948161ac0 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe93ffff640 1 -- 172.21.15.156:0/429272635 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944034660 con 0x7fe948161ac0 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 -- 172.21.15.156:0/429272635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ac0 msgr2=0x7fe9480cf060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/429272635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ac0 0x7fe9480cf060 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe94808ce30 tx=0x7fe94402bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 -- 172.21.15.156:0/429272635 shutdown_connections 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/429272635 >> v2:172.21.15.203:3300/0 conn(0x7fe9480cf5a0 0x7fe9480d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/429272635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ac0 0x7fe9480cf060 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 -- 172.21.15.156:0/429272635 >> 172.21.15.156:0/429272635 conn(0x7fe94808bfd0 msgr2=0x7fe9480d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 -- 172.21.15.156:0/429272635 shutdown_connections 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.327+0000 7fe94f5d4640 1 -- 172.21.15.156:0/429272635 wait complete. 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.328+0000 7fe94f5d4640 1 Processor -- start 2024-03-29T04:09:51.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.328+0000 7fe94f5d4640 1 -- start start 2024-03-29T04:09:51.310 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94f5d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe9480cf5a0 0x7fe948155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.310 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94f5d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948161ac0 0x7fe9480cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.310 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94f5d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 0x7fe9480b8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94f5d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe9480b72a0 con 0x7fe948161ea0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94f5d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9480d2020 con 0x7fe948161ac0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94f5d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe9480b7420 con 0x7fe9480cf5a0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 0x7fe9480b8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 0x7fe9480b8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40026/0 (socket says 172.21.15.156:40026) 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 -- 172.21.15.156:0/1982300451 learned_addr learned my addr 172.21.15.156:0/1982300451 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 -- 172.21.15.156:0/1982300451 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe9480cf5a0 msgr2=0x7fe948155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe9480cf5a0 0x7fe948155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 -- 172.21.15.156:0/1982300451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948161ac0 msgr2=0x7fe9480cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948161ac0 0x7fe9480cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.329+0000 7fe94db71640 1 -- 172.21.15.156:0/1982300451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9480cdb90 con 0x7fe948161ea0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.330+0000 7fe94db71640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 0x7fe9480b8650 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c410 tx=0x7fe94000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.330+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9400074e0 con 0x7fe948161ea0 2024-03-29T04:09:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.330+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9400047e0 con 0x7fe948161ea0 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.330+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9480cde70 con 0x7fe948161ea0 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.330+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940004c10 con 0x7fe948161ea0 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.330+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9480cd8a0 con 0x7fe948161ea0 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.331+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9480c7f20 con 0x7fe948161ea0 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.331+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe940011070 con 0x7fe948161ea0 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.332+0000 7fe93dffb640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe92c06c530 0x7fe92c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.332+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fe94008afb0 con 0x7fe948161ea0 2024-03-29T04:09:51.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.332+0000 7fe94d370640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe92c06c530 0x7fe92c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.332+0000 7fe94d370640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe92c06c530 0x7fe92c06e9f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe944036000 tx=0x7fe94402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.336+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe9480c7f20 con 0x7fe948161ea0 2024-03-29T04:09:51.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.512+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe948068610 con 0x7fe948161ea0 2024-03-29T04:09:51.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.513+0000 7fe93dffb640 1 -- 172.21.15.156:0/1982300451 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) ==== 84+0+65 (secure 0 0 0) 0x7fe94001a2b0 con 0x7fe948161ea0 2024-03-29T04:09:51.492 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:51.492 INFO:teuthology.orchestra.run.smithi156.stdout:[ 2024-03-29T04:09:51.493 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-03-29T04:09:51.493 INFO:teuthology.orchestra.run.smithi156.stdout: "poolnum": 1, 2024-03-29T04:09:51.493 INFO:teuthology.orchestra.run.smithi156.stdout: "poolname": ".mgr" 2024-03-29T04:09:51.493 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:09:51.493 INFO:teuthology.orchestra.run.smithi156.stdout:] 2024-03-29T04:09:51.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe92c06c530 msgr2=0x7fe92c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe92c06c530 0x7fe92c06e9f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe944036000 tx=0x7fe94402e000 comp rx=0 tx=0).stop 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 msgr2=0x7fe9480b8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 0x7fe9480b8650 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c410 tx=0x7fe94000c8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 shutdown_connections 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe92c06c530 0x7fe92c06e9f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe948161ea0 0x7fe9480b8650 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948161ac0 0x7fe9480cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 --2- 172.21.15.156:0/1982300451 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe9480cf5a0 0x7fe948155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 >> 172.21.15.156:0/1982300451 conn(0x7fe94808bfd0 msgr2=0x7fe9480b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 shutdown_connections 2024-03-29T04:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.515+0000 7fe94f5d4640 1 -- 172.21.15.156:0/1982300451 wait complete. 2024-03-29T04:09:51.505 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T04:09:51.505 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T04:09:51.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 Processor -- start 2024-03-29T04:09:51.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 -- start start 2024-03-29T04:09:51.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff9a8071fd0 0x7ff9a815eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a806f090 0x7ff9a80b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a8159030 con 0x7ff9a80d0470 2024-03-29T04:09:51.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a815faf0 con 0x7ff9a8071fd0 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.631+0000 7ff9acc1b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a815f250 con 0x7ff9a806f090 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff9a8071fd0 0x7ff9a815eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff9a8071fd0 0x7ff9a815eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35368/0 (socket says 172.21.15.156:35368) 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 -- 172.21.15.156:0/987834313 learned_addr learned my addr 172.21.15.156:0/987834313 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff9a6d9d640 1 --2- 172.21.15.156:0/987834313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a806f090 0x7ff9a80b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 -- 172.21.15.156:0/987834313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a806f090 msgr2=0x7ff9a80b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 --2- 172.21.15.156:0/987834313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a806f090 0x7ff9a80b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 -- 172.21.15.156:0/987834313 >> v1:172.21.15.203:6789/0 conn(0x7ff9a80d0470 legacy=0x7ff9a80d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.632+0000 7ff99ffff640 1 -- 172.21.15.156:0/987834313 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a815f530 con 0x7ff9a8071fd0 2024-03-29T04:09:51.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff99ffff640 1 --2- 172.21.15.156:0/987834313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a8071fd0 0x7ff9a815eb70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff990009870 tx=0x7ff99002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315ac1bdb560af33 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9a5d9b640 1 -- 172.21.15.156:0/987834313 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff990035020 con 0x7ff9a8071fd0 2024-03-29T04:09:51.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9a5d9b640 1 -- 172.21.15.156:0/987834313 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff990031640 con 0x7ff9a8071fd0 2024-03-29T04:09:51.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9a5d9b640 1 -- 172.21.15.156:0/987834313 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99002d040 con 0x7ff9a8071fd0 2024-03-29T04:09:51.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 -- 172.21.15.156:0/987834313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a8071fd0 msgr2=0x7ff9a815eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/987834313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a8071fd0 0x7ff9a815eb70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff990009870 tx=0x7ff99002b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 -- 172.21.15.156:0/987834313 shutdown_connections 2024-03-29T04:09:51.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/987834313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a806f090 0x7ff9a80b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/987834313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a8071fd0 0x7ff9a815eb70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 -- 172.21.15.156:0/987834313 >> 172.21.15.156:0/987834313 conn(0x7ff9a80b21e0 msgr2=0x7ff9a80b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:51.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 -- 172.21.15.156:0/987834313 shutdown_connections 2024-03-29T04:09:51.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.633+0000 7ff9acc1b640 1 -- 172.21.15.156:0/987834313 wait complete. 2024-03-29T04:09:51.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 Processor -- start 2024-03-29T04:09:51.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 -- start start 2024-03-29T04:09:51.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 0x7ff9a814a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a8071fd0 0x7ff9a8153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9a80d0470 0x7ff9a8154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a80b3910 con 0x7ff9a806f090 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a806f540 con 0x7ff9a80d0470 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9acc1b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a80b3a90 con 0x7ff9a8071fd0 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9a659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 0x7ff9a814a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9a659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 0x7ff9a814a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35376/0 (socket says 172.21.15.156:35376) 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.635+0000 7ff9a659c640 1 -- 172.21.15.156:0/1131305373 learned_addr learned my addr 172.21.15.156:0/1131305373 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a6d9d640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9a80d0470 0x7ff9a8154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff99ffff640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a8071fd0 0x7ff9a8153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a659c640 1 -- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a8071fd0 msgr2=0x7ff9a8153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a659c640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a8071fd0 0x7ff9a8153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a659c640 1 -- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9a80d0470 msgr2=0x7ff9a8154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a659c640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9a80d0470 0x7ff9a8154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a659c640 1 -- 172.21.15.156:0/1131305373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a814ace0 con 0x7ff9a806f090 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff99ffff640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a8071fd0 0x7ff9a8153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9a659c640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 0x7ff9a814a7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff99800c3f0 tx=0x7ff99800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980052d0 con 0x7ff9a806f090 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff998005490 con 0x7ff9a806f090 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a8153b60 con 0x7ff9a806f090 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998007260 con 0x7ff9a806f090 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.636+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a8155010 con 0x7ff9a806f090 2024-03-29T04:09:51.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.637+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a80b4930 con 0x7ff9a806f090 2024-03-29T04:09:51.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.638+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff998007400 con 0x7ff9a806f090 2024-03-29T04:09:51.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.638+0000 7ff99effd640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff98806c530 0x7ff98806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:51.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.638+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7ff99808a760 con 0x7ff9a806f090 2024-03-29T04:09:51.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.638+0000 7ff99ffff640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff98806c530 0x7ff98806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:51.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.638+0000 7ff99ffff640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff98806c530 0x7ff98806e9f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff990036000 tx=0x7ff99003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:51.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.642+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff9a80b4930 con 0x7ff9a806f090 2024-03-29T04:09:51.797 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:51.818+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff9a8068610 con 0x7ff9a806f090 2024-03-29T04:09:51.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.000+0000 7ff99effd640 1 -- 172.21.15.156:0/1131305373 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) ==== 138+0+0 (secure 0 0 0) 0x7ff9a8159030 con 0x7ff9a806f090 2024-03-29T04:09:51.979 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_metadata' created 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff98806c530 msgr2=0x7ff98806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff98806c530 0x7ff98806e9f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff990036000 tx=0x7ff99003c040 comp rx=0 tx=0).stop 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 msgr2=0x7ff9a814a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 0x7ff9a814a7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff99800c3f0 tx=0x7ff99800c8b0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 shutdown_connections 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff98806c530 0x7ff98806e9f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9a80d0470 0x7ff9a8154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff9a8071fd0 0x7ff9a8153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 --2- 172.21.15.156:0/1131305373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a806f090 0x7ff9a814a7a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 >> 172.21.15.156:0/1131305373 conn(0x7ff9a80b21e0 msgr2=0x7ff9a815fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 shutdown_connections 2024-03-29T04:09:51.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.002+0000 7ff9acc1b640 1 -- 172.21.15.156:0/1131305373 wait complete. 2024-03-29T04:09:51.997 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T04:09:52.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.126+0000 7fe46bc0d640 1 Processor -- start 2024-03-29T04:09:52.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.126+0000 7fe46bc0d640 1 -- start start 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46bc0d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0350 0x7fe46415c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46bc0d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe46415eec0 0x7fe46415f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46bc0d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe46415b8d0 con 0x7fe4640d0730 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46bc0d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe46415c8d0 con 0x7fe46415eec0 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46bc0d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe46415bf00 con 0x7fe4640d0350 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46a1aa640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe46415eec0 0x7fe46415f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe4691a8640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fe4640d0730 0x7fe46406d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44664/0 (socket says 172.21.15.156:44664) 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe4691a8640 1 -- 172.21.15.156:0/1573849234 learned_addr learned my addr 172.21.15.156:0/1573849234 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46a1aa640 1 -- 172.21.15.156:0/1573849234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0350 msgr2=0x7fe46415c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46a1aa640 1 --2- 172.21.15.156:0/1573849234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0350 0x7fe46415c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46a1aa640 1 -- 172.21.15.156:0/1573849234 >> v1:172.21.15.156:6789/0 conn(0x7fe4640d0730 legacy=0x7fe46406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe46a1aa640 1 -- 172.21.15.156:0/1573849234 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46406e120 con 0x7fe46415eec0 2024-03-29T04:09:52.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.127+0000 7fe4689a7640 1 -- 172.21.15.156:0/1573849234 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014003747 0 0) 0x7fe46415b8d0 con 0x7fe4640d0730 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.128+0000 7fe46a1aa640 1 --2- 172.21.15.156:0/1573849234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe46415eec0 0x7fe46415f350 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009d90 tx=0x7fe45c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5613cf7084e1e6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.128+0000 7fe4689a7640 1 -- 172.21.15.156:0/1573849234 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c039070 con 0x7fe46415eec0 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.128+0000 7fe4689a7640 1 -- 172.21.15.156:0/1573849234 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45c02cc50 con 0x7fe46415eec0 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.128+0000 7fe4689a7640 1 -- 172.21.15.156:0/1573849234 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c034440 con 0x7fe46415eec0 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.128+0000 7fe46bc0d640 1 -- 172.21.15.156:0/1573849234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe46415eec0 msgr2=0x7fe46415f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.128+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/1573849234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe46415eec0 0x7fe46415f350 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009d90 tx=0x7fe45c02bea0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.129+0000 7fe46bc0d640 1 -- 172.21.15.156:0/1573849234 shutdown_connections 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.129+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/1573849234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe46415eec0 0x7fe46415f350 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.129+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/1573849234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0350 0x7fe46415c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.129+0000 7fe46bc0d640 1 -- 172.21.15.156:0/1573849234 >> 172.21.15.156:0/1573849234 conn(0x7fe4640b1f30 msgr2=0x7fe4640b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:52.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.129+0000 7fe46bc0d640 1 -- 172.21.15.156:0/1573849234 shutdown_connections 2024-03-29T04:09:52.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.129+0000 7fe46bc0d640 1 -- 172.21.15.156:0/1573849234 wait complete. 2024-03-29T04:09:52.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 Processor -- start 2024-03-29T04:09:52.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 -- start start 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 0x7fe464146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0730 0x7fe4641f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe46415eec0 0x7fe4641568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe464073700 con 0x7fe4640d0350 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe46415bb90 con 0x7fe46415eec0 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.130+0000 7fe46bc0d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe4640d0bb0 con 0x7fe4640d0730 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 0x7fe464146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 0x7fe464146540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35402/0 (socket says 172.21.15.156:35402) 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 -- 172.21.15.156:0/2520643928 learned_addr learned my addr 172.21.15.156:0/2520643928 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4691a8640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0730 0x7fe4641f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:52.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 -- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0730 msgr2=0x7fe4641f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0730 0x7fe4641f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 -- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe46415eec0 msgr2=0x7fe4641568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe46415eec0 0x7fe4641568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 -- 172.21.15.156:0/2520643928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4640afbf0 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe4699a9640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 0x7fe464146540 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe46000b4d0 tx=0x7fe46000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460014070 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe460010870 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4640afe90 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460010c60 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.131+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe464156e20 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.132+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46404f9d0 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.133+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe460017070 con 0x7fe4640d0350 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.133+0000 7fe45a7fc640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe44806c530 0x7fe44806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:52.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.133+0000 7fe4691a8640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe44806c530 0x7fe44806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:52.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.133+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5227+0+0 (secure 0 0 0) 0x7fe46008b250 con 0x7fe4640d0350 2024-03-29T04:09:52.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.134+0000 7fe4691a8640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe44806c530 0x7fe44806e9f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe454007fd0 tx=0x7fe454009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:52.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.138+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe460059ae0 con 0x7fe4640d0350 2024-03-29T04:09:52.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:52.311+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe464072510 con 0x7fe4640d0350 2024-03-29T04:09:52.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.002+0000 7fe45a7fc640 1 -- 172.21.15.156:0/2520643928 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) ==== 170+0+0 (secure 0 0 0) 0x7fe4600593e0 con 0x7fe4640d0350 2024-03-29T04:09:52.982 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_data' created 2024-03-29T04:09:52.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe44806c530 msgr2=0x7fe44806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe44806c530 0x7fe44806e9f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe454007fd0 tx=0x7fe454009040 comp rx=0 tx=0).stop 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 msgr2=0x7fe464146540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 0x7fe464146540 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe46000b4d0 tx=0x7fe46000b990 comp rx=0 tx=0).stop 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 shutdown_connections 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe44806c530 0x7fe44806e9f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe46415eec0 0x7fe4641568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4640d0730 0x7fe4641f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 --2- 172.21.15.156:0/2520643928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640d0350 0x7fe464146540 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 >> 172.21.15.156:0/2520643928 conn(0x7fe4640b1f30 msgr2=0x7fe46409c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 shutdown_connections 2024-03-29T04:09:52.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.005+0000 7fe46bc0d640 1 -- 172.21.15.156:0/2520643928 wait complete. 2024-03-29T04:09:53.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T04:09:53.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.134+0000 7f8ffda55640 1 Processor -- start 2024-03-29T04:09:53.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.134+0000 7f8ffda55640 1 -- start start 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ffda55640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff80d2120 0x7f8ff806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ffda55640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8ff806cb10 0x7f8ff80cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ffda55640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ff80d2df0 con 0x7f8ff815e920 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ffda55640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8ff80cf7a0 con 0x7f8ff806cb10 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ffda55640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff80cf190 con 0x7f8ff80d2120 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ff77fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8ff806cb10 0x7f8ff80cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ff77fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8ff806cb10 0x7f8ff80cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35406/0 (socket says 172.21.15.156:35406) 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8ff77fe640 1 -- 172.21.15.156:0/1895476950 learned_addr learned my addr 172.21.15.156:0/1895476950 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.135+0000 7f8fee7fc640 1 --2- 172.21.15.156:0/1895476950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff80d2120 0x7f8ff806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff77fe640 1 -- 172.21.15.156:0/1895476950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff80d2120 msgr2=0x7f8ff806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:53.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff77fe640 1 --2- 172.21.15.156:0/1895476950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff80d2120 0x7f8ff806c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:53.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff77fe640 1 -- 172.21.15.156:0/1895476950 >> v1:172.21.15.203:6789/0 conn(0x7f8ff815e920 legacy=0x7f8ff80d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:53.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff77fe640 1 -- 172.21.15.156:0/1895476950 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff80d3080 con 0x7f8ff806cb10 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff77fe640 1 --2- 172.21.15.156:0/1895476950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff806cb10 0x7f8ff80cfc30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009e50 tx=0x7f8ff002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fddd9a75c835be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff67fc640 1 -- 172.21.15.156:0/1895476950 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff002ca70 con 0x7f8ff806cb10 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff67fc640 1 -- 172.21.15.156:0/1895476950 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff002cc10 con 0x7f8ff806cb10 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.136+0000 7f8ff67fc640 1 -- 172.21.15.156:0/1895476950 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00343f0 con 0x7f8ff806cb10 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 -- 172.21.15.156:0/1895476950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff806cb10 msgr2=0x7f8ff80cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 --2- 172.21.15.156:0/1895476950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff806cb10 0x7f8ff80cfc30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009e50 tx=0x7f8ff002bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 -- 172.21.15.156:0/1895476950 shutdown_connections 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 --2- 172.21.15.156:0/1895476950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff806cb10 0x7f8ff80cfc30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 --2- 172.21.15.156:0/1895476950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff80d2120 0x7f8ff806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 -- 172.21.15.156:0/1895476950 >> 172.21.15.156:0/1895476950 conn(0x7f8ff809d7a0 msgr2=0x7f8ff80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:53.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 -- 172.21.15.156:0/1895476950 shutdown_connections 2024-03-29T04:09:53.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.137+0000 7f8ffda55640 1 -- 172.21.15.156:0/1895476950 wait complete. 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.138+0000 7f8ffda55640 1 Processor -- start 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.138+0000 7f8ffda55640 1 -- start start 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ffda55640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff806cb10 0x7f8ff81518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ffda55640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 0x7f8ff80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ffda55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff815e920 0x7f8ff80b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ffda55640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff80d34e0 con 0x7f8ff80d2120 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ffda55640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff80d3360 con 0x7f8ff815e920 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ffda55640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff80d0170 con 0x7f8ff806cb10 2024-03-29T04:09:53.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 0x7f8ff80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 0x7f8ff80b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35416/0 (socket says 172.21.15.156:35416) 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 -- 172.21.15.156:0/14599115 learned_addr learned my addr 172.21.15.156:0/14599115 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ff6ffd640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff806cb10 0x7f8ff81518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8ff77fe640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff815e920 0x7f8ff80b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 -- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff806cb10 msgr2=0x7f8ff81518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff806cb10 0x7f8ff81518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 -- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff815e920 msgr2=0x7f8ff80b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.139+0000 7f8fee7fc640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff815e920 0x7f8ff80b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8fee7fc640 1 -- 172.21.15.156:0/14599115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff8151e00 con 0x7f8ff80d2120 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8fee7fc640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 0x7f8ff80b3e90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c7f0 tx=0x7f8fe000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0015070 con 0x7f8ff80d2120 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe0004240 con 0x7f8ff80d2120 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff80b4800 con 0x7f8ff80d2120 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0007620 con 0x7f8ff80d2120 2024-03-29T04:09:53.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.140+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff80b4dc0 con 0x7f8ff80d2120 2024-03-29T04:09:53.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.141+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff80a2480 con 0x7f8ff80d2120 2024-03-29T04:09:53.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.141+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8fe00079d0 con 0x7f8ff80d2120 2024-03-29T04:09:53.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.142+0000 7f8ff4ff9640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8fd806c530 0x7f8fd806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:53.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.142+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5609+0+0 (secure 0 0 0) 0x7f8fe008bb50 con 0x7f8ff80d2120 2024-03-29T04:09:53.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.142+0000 7f8ff6ffd640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8fd806c530 0x7f8fd806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:53.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.142+0000 7f8ff6ffd640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8fd806c530 0x7f8fd806e9f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8006fd0 tx=0x7f8fe8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:53.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.146+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8fe0059c60 con 0x7f8ff80d2120 2024-03-29T04:09:53.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:53.359+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8ff804f9d0 con 0x7f8ff80d2120 2024-03-29T04:09:54.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.030+0000 7f8ff4ff9640 1 -- 172.21.15.156:0/14599115 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7f8fe005df60 con 0x7f8ff80d2120 2024-03-29T04:09:54.010 INFO:teuthology.orchestra.run.smithi156.stderr:new fs with metadata pool 4 and data pool 5 2024-03-29T04:09:54.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.039+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8fd806c530 msgr2=0x7f8fd806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.039+0000 7f8ffda55640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8fd806c530 0x7f8fd806e9f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8006fd0 tx=0x7f8fe8009060 comp rx=0 tx=0).stop 2024-03-29T04:09:54.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.039+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 msgr2=0x7f8ff80b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.039+0000 7f8ffda55640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 0x7f8ff80b3e90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c7f0 tx=0x7f8fe000ccb0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.040+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 shutdown_connections 2024-03-29T04:09:54.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.040+0000 7f8ffda55640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8fd806c530 0x7f8fd806e9f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.040+0000 7f8ffda55640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff815e920 0x7f8ff80b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.040+0000 7f8ffda55640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ff80d2120 0x7f8ff80b3e90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.040+0000 7f8ffda55640 1 --2- 172.21.15.156:0/14599115 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff806cb10 0x7f8ff81518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.040+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 >> 172.21.15.156:0/14599115 conn(0x7f8ff809d7a0 msgr2=0x7f8ff80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:54.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.041+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 shutdown_connections 2024-03-29T04:09:54.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.042+0000 7f8ffda55640 1 -- 172.21.15.156:0/14599115 wait complete. 2024-03-29T04:09:54.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:54.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.159+0000 7fa50ebf2640 1 Processor -- start 2024-03-29T04:09:54.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.159+0000 7fa50ebf2640 1 -- start start 2024-03-29T04:09:54.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.159+0000 7fa50ebf2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa50806dbc0 0x7fa508169350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.159+0000 7fa50ebf2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5080cfa20 con 0x7fa5080d42c0 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.159+0000 7fa50ebf2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5080cfd00 con 0x7fa50815e730 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.159+0000 7fa50ebf2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa50806c720 con 0x7fa50806dbc0 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa50d18f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa50806dbc0 0x7fa508169350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa50d18f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa50806dbc0 0x7fa508169350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35424/0 (socket says 172.21.15.156:35424) 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa507fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa5080d42c0 0x7fa5080d4740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44670/0 (socket says 172.21.15.156:44670) 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa50d18f640 1 -- 172.21.15.156:0/1291691826 learned_addr learned my addr 172.21.15.156:0/1291691826 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa50d18f640 1 -- 172.21.15.156:0/1291691826 >> v1:172.21.15.203:6789/0 conn(0x7fa50815e730 legacy=0x7fa5080ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa50d18f640 1 -- 172.21.15.156:0/1291691826 >> v1:172.21.15.156:6789/0 conn(0x7fa5080d42c0 legacy=0x7fa5080d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa50d18f640 1 -- 172.21.15.156:0/1291691826 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5080cffe0 con 0x7fa50806dbc0 2024-03-29T04:09:54.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.160+0000 7fa5077fe640 1 -- 172.21.15.156:0/1291691826 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1165655226 0 0) 0x7fa5080cfa20 con 0x7fa5080d42c0 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50d18f640 1 --2- 172.21.15.156:0/1291691826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50806dbc0 0x7fa508169350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa500009e50 tx=0x7fa50002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981ef4b0502b741a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa5077fe640 1 -- 172.21.15.156:0/1291691826 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50002cb20 con 0x7fa50806dbc0 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa5077fe640 1 -- 172.21.15.156:0/1291691826 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa50002ccc0 con 0x7fa50806dbc0 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa5077fe640 1 -- 172.21.15.156:0/1291691826 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa500034610 con 0x7fa50806dbc0 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1291691826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50806dbc0 msgr2=0x7fa508169350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1291691826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50806dbc0 0x7fa508169350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa500009e50 tx=0x7fa50002bf80 comp rx=0 tx=0).stop 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1291691826 shutdown_connections 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1291691826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50806dbc0 0x7fa508169350 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1291691826 >> 172.21.15.156:0/1291691826 conn(0x7fa50808bfd0 msgr2=0x7fa50809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1291691826 shutdown_connections 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.161+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1291691826 wait complete. 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.162+0000 7fa50ebf2640 1 Processor -- start 2024-03-29T04:09:54.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 -- start start 2024-03-29T04:09:54.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa50806dbc0 0x7fa508153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5080d42c0 0x7fa508160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 0x7fa50815bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa50815eb10 con 0x7fa50815e730 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa5080d2220 con 0x7fa5080d42c0 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.163+0000 7fa50ebf2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa50815ec90 con 0x7fa50806dbc0 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 0x7fa50815bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 0x7fa50815bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35436/0 (socket says 172.21.15.156:35436) 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 -- 172.21.15.156:0/1777663063 learned_addr learned my addr 172.21.15.156:0/1777663063 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50c98e640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa50806dbc0 0x7fa508153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa507fff640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5080d42c0 0x7fa508160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 -- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa50806dbc0 msgr2=0x7fa508153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa50806dbc0 0x7fa508153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 -- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5080d42c0 msgr2=0x7fa508160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5080d42c0 0x7fa508160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 -- 172.21.15.156:0/1777663063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5081f4940 con 0x7fa50815e730 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50d18f640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 0x7fa50815bf60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa500036040 tx=0x7fa500004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa500043020 con 0x7fa50815e730 2024-03-29T04:09:54.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa50003e450 con 0x7fa50815e730 2024-03-29T04:09:54.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5081f4c20 con 0x7fa50815e730 2024-03-29T04:09:54.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa500009510 con 0x7fa50815e730 2024-03-29T04:09:54.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.164+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5081f51e0 con 0x7fa50815e730 2024-03-29T04:09:54.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.165+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa508068610 con 0x7fa50815e730 2024-03-29T04:09:54.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.166+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa500046020 con 0x7fa50815e730 2024-03-29T04:09:54.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.166+0000 7fa5057fa640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa4ec06c560 0x7fa4ec06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.166+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fa5000adcc0 con 0x7fa50815e730 2024-03-29T04:09:54.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.166+0000 7fa50c98e640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa4ec06c560 0x7fa4ec06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.167+0000 7fa50c98e640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa4ec06c560 0x7fa4ec06ea20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5081f5bf0 tx=0x7fa4fc009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.170+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa50004d070 con 0x7fa50815e730 2024-03-29T04:09:54.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.437+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5081f53e0 con 0x7fa50815e730 2024-03-29T04:09:54.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.438+0000 7fa5057fa640 1 -- 172.21.15.156:0/1777663063 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16706 (secure 0 0 0) 0x7fa5081f53e0 con 0x7fa50815e730 2024-03-29T04:09:54.417 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:54.418 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":33,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:53.360425+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:54.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa4ec06c560 msgr2=0x7fa4ec06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa4ec06c560 0x7fa4ec06ea20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5081f5bf0 tx=0x7fa4fc009060 comp rx=0 tx=0).stop 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 msgr2=0x7fa50815bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 0x7fa50815bf60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa500036040 tx=0x7fa500004a40 comp rx=0 tx=0).stop 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 shutdown_connections 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa4ec06c560 0x7fa4ec06ea20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa50815e730 0x7fa50815bf60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5080d42c0 0x7fa508160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 --2- 172.21.15.156:0/1777663063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa50806dbc0 0x7fa508153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 >> 172.21.15.156:0/1777663063 conn(0x7fa50808bfd0 msgr2=0x7fa5080d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 shutdown_connections 2024-03-29T04:09:54.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.440+0000 7fa50ebf2640 1 -- 172.21.15.156:0/1777663063 wait complete. 2024-03-29T04:09:54.431 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:54.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.553+0000 7f70b3c14640 1 Processor -- start 2024-03-29T04:09:54.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.553+0000 7f70b3c14640 1 -- start start 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b3c14640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 0x7f70ac162e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b3c14640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f70ac0b1bb0 0x7f70ac0d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b3c14640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ac15fcf0 con 0x7f70ac162340 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b3c14640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f70ac0b2040 con 0x7f70ac0b1bb0 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b3c14640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f70ac0cd9c0 con 0x7f70ac0b31e0 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b19b0640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f70ac162340 0x7f70ac162720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44674/0 (socket says 172.21.15.156:44674) 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b19b0640 1 -- 172.21.15.156:0/1130704778 learned_addr learned my addr 172.21.15.156:0/1130704778 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1130704778 >> v2:172.21.15.203:3300/0 conn(0x7f70ac0b1bb0 0x7f70ac0d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b11af640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 0x7f70ac162e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366712007 0 0) 0x7f70ac15fcf0 con 0x7f70ac162340 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7098003520 con 0x7f70ac162340 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4149089104 0 0) 0x7f7098003520 con 0x7f70ac162340 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 msgr2=0x7f70ac162e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 0x7f70ac162e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 >> v2:172.21.15.203:3300/0 conn(0x7f70ac0b1bb0 msgr2=0x7f70ac0d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 --2- 172.21.15.156:0/1130704778 >> v2:172.21.15.203:3300/0 conn(0x7f70ac0b1bb0 0x7f70ac0d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac0cdc10 con 0x7f70ac162340 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f70a8001670 con 0x7f70ac162340 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.554+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 >> v1:172.21.15.156:6789/0 conn(0x7f70ac162340 legacy=0x7f70ac162720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b09ae640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 0x7f70980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b09ae640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7098003f30 0x7f7098004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b09ae640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7098004d40 0x7f70980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac0cd9c0 con 0x7f7098004d40 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f709800b9c0 con 0x7f7098003f30 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f709800bb90 con 0x7f70ac0b31e0 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7098004d40 0x7f70980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b11af640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7098003f30 0x7f7098004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b19b0640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 0x7f70980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b21b1640 1 -- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 msgr2=0x7f70980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac0b31e0 0x7f70980039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b21b1640 1 -- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7098003f30 msgr2=0x7f7098004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7098003f30 0x7f7098004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.555+0000 7f70b21b1640 1 -- 172.21.15.156:0/1130704778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7098003520 con 0x7f7098004d40 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7098004d40 0x7f70980051c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f70a400e170 tx=0x7f7098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68936994a9a535f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4039070 con 0x7f7098004d40 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a402cc20 con 0x7f7098004d40 2024-03-29T04:09:54.538 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b09ae640 1 -- 172.21.15.156:0/1130704778 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4009c10 con 0x7f7098004d40 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b3c14640 1 -- 172.21.15.156:0/1130704778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7098004d40 msgr2=0x7f70980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1130704778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7098004d40 0x7f70980051c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f70a400e170 tx=0x7f7098005830 comp rx=0 tx=0).stop 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b21b1640 1 -- 172.21.15.156:0/1130704778 reap_dead start 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b3c14640 1 -- 172.21.15.156:0/1130704778 shutdown_connections 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.556+0000 7f70b3c14640 1 -- 172.21.15.156:0/1130704778 >> 172.21.15.156:0/1130704778 conn(0x7f70ac0b3d90 msgr2=0x7f70ac0b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.557+0000 7f70b3c14640 1 -- 172.21.15.156:0/1130704778 shutdown_connections 2024-03-29T04:09:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.557+0000 7f70b3c14640 1 -- 172.21.15.156:0/1130704778 wait complete. 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 Processor -- start 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 -- start start 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac15a690 0x7f70ac15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac15aa70 0x7f70ac0b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 0x7f70ac0c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac0d01f0 con 0x7f70ac0c7d90 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac0cff10 con 0x7f70ac15aa70 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.558+0000 7f70b3c14640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f70ac0ce1d0 con 0x7f70ac15a690 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 0x7f70ac0c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 0x7f70ac0c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35454/0 (socket says 172.21.15.156:35454) 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 -- 172.21.15.156:0/1518693292 learned_addr learned my addr 172.21.15.156:0/1518693292 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b19b0640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac15a690 0x7f70ac15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b11af640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac15aa70 0x7f70ac0b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 -- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac15a690 msgr2=0x7f70ac15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac15a690 0x7f70ac15bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 -- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac15aa70 msgr2=0x7f70ac0b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac15aa70 0x7f70ac0b7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 -- 172.21.15.156:0/1518693292 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac0b64c0 con 0x7f70ac0c7d90 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b21b1640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 0x7f70ac0c8240 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f70a4037040 tx=0x7f70a4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4039040 con 0x7f70ac0c7d90 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a402ca20 con 0x7f70ac0c7d90 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ac0c6fd0 con 0x7f70ac0c7d90 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4009c10 con 0x7f70ac0c7d90 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.559+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ac0c7590 con 0x7f70ac0c7d90 2024-03-29T04:09:54.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.560+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70ac068d20 con 0x7f70ac0c7d90 2024-03-29T04:09:54.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.561+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f70a4048030 con 0x7f70ac0c7d90 2024-03-29T04:09:54.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.561+0000 7f70a27fc640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f709006c530 0x7f709006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.561+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f70a40adee0 con 0x7f70ac0c7d90 2024-03-29T04:09:54.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.561+0000 7f70b19b0640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f709006c530 0x7f709006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.562+0000 7f70b19b0640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f709006c530 0x7f709006e9f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f70a8004840 tx=0x7f70a8007970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.565+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f70a404f060 con 0x7f70ac0c7d90 2024-03-29T04:09:54.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.811+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70ac0b4940 con 0x7f70ac0c7d90 2024-03-29T04:09:54.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.812+0000 7f70a27fc640 1 -- 172.21.15.156:0/1518693292 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16706 (secure 0 0 0) 0x7f70a402cbf0 con 0x7f70ac0c7d90 2024-03-29T04:09:54.792 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:54.792 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":33,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:53.360425+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:54.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f709006c530 msgr2=0x7f709006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f709006c530 0x7f709006e9f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f70a8004840 tx=0x7f70a8007970 comp rx=0 tx=0).stop 2024-03-29T04:09:54.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 msgr2=0x7f70ac0c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 0x7f70ac0c8240 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f70a4037040 tx=0x7f70a4009750 comp rx=0 tx=0).stop 2024-03-29T04:09:54.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 shutdown_connections 2024-03-29T04:09:54.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f709006c530 0x7f709006e9f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70ac0c7d90 0x7f70ac0c8240 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac15aa70 0x7f70ac0b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 --2- 172.21.15.156:0/1518693292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f70ac15a690 0x7f70ac15bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 >> 172.21.15.156:0/1518693292 conn(0x7f70ac0b3d90 msgr2=0x7f70ac0d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.814+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 shutdown_connections 2024-03-29T04:09:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.815+0000 7f70b3c14640 1 -- 172.21.15.156:0/1518693292 wait complete. 2024-03-29T04:09:54.805 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T04:09:54.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.930+0000 7f5733740640 1 Processor -- start 2024-03-29T04:09:54.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.931+0000 7f5733740640 1 -- start start 2024-03-29T04:09:54.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.931+0000 7f5733740640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f572c06e670 0x7f572c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.931+0000 7f5733740640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c15ef90 0x7f572c15b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.931+0000 7f5733740640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572c0727d0 con 0x7f572c06e290 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.931+0000 7f5733740640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f572c072ab0 con 0x7f572c06e670 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.931+0000 7f5733740640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f572c072d90 con 0x7f572c15ef90 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5730cdb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f572c06e670 0x7f572c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5731cdd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c15ef90 0x7f572c15b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f57314dc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f572c06e290 0x7f572c0b3050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:40940/0 (socket says 172.21.15.156:40940) 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f57314dc640 1 -- 172.21.15.156:0/2437078841 learned_addr learned my addr 172.21.15.156:0/2437078841 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5730cdb640 1 -- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c15ef90 msgr2=0x7f572c15b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5730cdb640 1 --2- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c15ef90 0x7f572c15b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5730cdb640 1 -- 172.21.15.156:0/2437078841 >> v1:172.21.15.203:6789/0 conn(0x7f572c06e290 legacy=0x7f572c0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5730cdb640 1 -- 172.21.15.156:0/2437078841 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c073070 con 0x7f572c06e670 2024-03-29T04:09:54.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5723fff640 1 -- 172.21.15.156:0/2437078841 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639494026 0 0) 0x7f572c0727d0 con 0x7f572c06e290 2024-03-29T04:09:54.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.932+0000 7f5731cdd640 1 --2- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c15ef90 0x7f572c15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:54.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.933+0000 7f5730cdb640 1 --2- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e670 0x7f572c072140 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f571c0097a0 tx=0x7f571c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7632ce8e2a64622e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.933+0000 7f5723fff640 1 -- 172.21.15.156:0/2437078841 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c0315a0 con 0x7f572c06e670 2024-03-29T04:09:54.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.933+0000 7f5723fff640 1 -- 172.21.15.156:0/2437078841 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571c037070 con 0x7f572c06e670 2024-03-29T04:09:54.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.933+0000 7f5723fff640 1 -- 172.21.15.156:0/2437078841 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c031960 con 0x7f572c06e670 2024-03-29T04:09:54.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.933+0000 7f5733740640 1 -- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e670 msgr2=0x7f572c072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.933+0000 7f5733740640 1 --2- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e670 0x7f572c072140 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f571c0097a0 tx=0x7f571c02b840 comp rx=0 tx=0).stop 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.934+0000 7f5733740640 1 -- 172.21.15.156:0/2437078841 shutdown_connections 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.934+0000 7f5733740640 1 --2- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c15ef90 0x7f572c15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.934+0000 7f5733740640 1 --2- 172.21.15.156:0/2437078841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e670 0x7f572c072140 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.934+0000 7f5733740640 1 -- 172.21.15.156:0/2437078841 >> 172.21.15.156:0/2437078841 conn(0x7f572c0b1f30 msgr2=0x7f572c0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.934+0000 7f5733740640 1 -- 172.21.15.156:0/2437078841 shutdown_connections 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.934+0000 7f5733740640 1 -- 172.21.15.156:0/2437078841 wait complete. 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 Processor -- start 2024-03-29T04:09:54.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 -- start start 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e290 0x7f572c155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c06e670 0x7f572c0b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 0x7f572c15d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f572c073870 con 0x7f572c15ef90 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f572c15bd90 con 0x7f572c06e290 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.935+0000 7f5733740640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f572c0739f0 con 0x7f572c06e670 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 0x7f572c15d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 0x7f572c15d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35464/0 (socket says 172.21.15.156:35464) 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 -- 172.21.15.156:0/622831777 learned_addr learned my addr 172.21.15.156:0/622831777 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5730cdb640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c06e670 0x7f572c0b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f57314dc640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e290 0x7f572c155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 -- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c06e670 msgr2=0x7f572c0b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c06e670 0x7f572c0b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 -- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e290 msgr2=0x7f572c155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e290 0x7f572c155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 -- 172.21.15.156:0/622831777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c15c9c0 con 0x7f572c15ef90 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5731cdd640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 0x7f572c15d660 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5724009e00 tx=0x7f57240075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724003990 con 0x7f572c15ef90 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5724003b30 con 0x7f572c15ef90 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c15cca0 con 0x7f572c15ef90 2024-03-29T04:09:54.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.936+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724004020 con 0x7f572c15ef90 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.937+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c15d260 con 0x7f572c15ef90 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.938+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572c09e3b0 con 0x7f572c15ef90 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.938+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5724005d10 con 0x7f572c15ef90 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.938+0000 7f5721ffb640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f571006c530 0x7f571006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.939+0000 7f57314dc640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f571006c530 0x7f571006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.939+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f572408b6b0 con 0x7f572c15ef90 2024-03-29T04:09:54.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.939+0000 7f57314dc640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f571006c530 0x7f571006e9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5728000f60 tx=0x7f5728007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:54.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:54.943+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5724022020 con 0x7f572c15ef90 2024-03-29T04:09:55.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:55.174+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f572c0af9a0 con 0x7f572c15ef90 2024-03-29T04:09:55.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:09:55.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:09:56.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.037+0000 7f5721ffb640 1 -- 172.21.15.156:0/622831777 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7f5724011070 con 0x7f572c15ef90 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f571006c530 msgr2=0x7f571006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f571006c530 0x7f571006e9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5728000f60 tx=0x7f5728007d50 comp rx=0 tx=0).stop 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 msgr2=0x7f572c15d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 0x7f572c15d660 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5724009e00 tx=0x7f57240075c0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 shutdown_connections 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f571006c530 0x7f571006e9f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f572c15ef90 0x7f572c15d660 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f572c06e670 0x7f572c0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 --2- 172.21.15.156:0/622831777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c06e290 0x7f572c155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 >> 172.21.15.156:0/622831777 conn(0x7f572c0b1f30 msgr2=0x7f572c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 shutdown_connections 2024-03-29T04:09:56.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.039+0000 7f5733740640 1 -- 172.21.15.156:0/622831777 wait complete. 2024-03-29T04:09:56.029 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T04:09:56.030 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T04:09:56.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.155+0000 7fa7105f0640 1 Processor -- start 2024-03-29T04:09:56.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.155+0000 7fa7105f0640 1 -- start start 2024-03-29T04:09:56.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa7105f0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815e920 0x7fa7080d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:56.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa7105f0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa7080d2120 0x7fa70806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa7105f0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7080cf190 con 0x7fa70806cb10 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa7105f0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa7080cf7a0 con 0x7fa7080d2120 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa7105f0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa7080d3080 con 0x7fa70815e920 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa70db8b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa7080d2120 0x7fa70806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa70eb8d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa70806cb10 0x7fa7080cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44678/0 (socket says 172.21.15.156:44678) 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.156+0000 7fa70eb8d640 1 -- 172.21.15.156:0/4047245705 learned_addr learned my addr 172.21.15.156:0/4047245705 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.157+0000 7fa70e38c640 1 --2- 172.21.15.156:0/4047245705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815e920 0x7fa7080d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.157+0000 7fa70db8b640 1 -- 172.21.15.156:0/4047245705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815e920 msgr2=0x7fa7080d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.157+0000 7fa70db8b640 1 --2- 172.21.15.156:0/4047245705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815e920 0x7fa7080d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.157+0000 7fa70db8b640 1 -- 172.21.15.156:0/4047245705 >> v1:172.21.15.156:6789/0 conn(0x7fa70806cb10 legacy=0x7fa7080cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.157+0000 7fa70d38a640 1 -- 172.21.15.156:0/4047245705 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946559679 0 0) 0x7fa7080cf190 con 0x7fa70806cb10 2024-03-29T04:09:56.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.157+0000 7fa70db8b640 1 -- 172.21.15.156:0/4047245705 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7080cf430 con 0x7fa7080d2120 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.158+0000 7fa70db8b640 1 --2- 172.21.15.156:0/4047245705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080d2120 0x7fa70806c5d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80097a0 tx=0x7fa6f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed68606f09a1d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.158+0000 7fa70d38a640 1 -- 172.21.15.156:0/4047245705 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8035020 con 0x7fa7080d2120 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.158+0000 7fa70d38a640 1 -- 172.21.15.156:0/4047245705 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f8031830 con 0x7fa7080d2120 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.158+0000 7fa70d38a640 1 -- 172.21.15.156:0/4047245705 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8031c00 con 0x7fa7080d2120 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 -- 172.21.15.156:0/4047245705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080d2120 msgr2=0x7fa70806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4047245705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080d2120 0x7fa70806c5d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80097a0 tx=0x7fa6f802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 -- 172.21.15.156:0/4047245705 shutdown_connections 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4047245705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080d2120 0x7fa70806c5d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4047245705 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa70815e920 0x7fa7080d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 -- 172.21.15.156:0/4047245705 >> 172.21.15.156:0/4047245705 conn(0x7fa70809d7a0 msgr2=0x7fa7080d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 -- 172.21.15.156:0/4047245705 shutdown_connections 2024-03-29T04:09:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.159+0000 7fa7105f0640 1 -- 172.21.15.156:0/4047245705 wait complete. 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.160+0000 7fa7105f0640 1 Processor -- start 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.160+0000 7fa7105f0640 1 -- start start 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 0x7fa7080cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080d2120 0x7fa7081e9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa70815e920 0x7fa7081e9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7080d04c0 con 0x7fa70806cb10 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7080d0340 con 0x7fa70815e920 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa7080d2c20 con 0x7fa7080d2120 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 0x7fa7080cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 0x7fa7080cc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35482/0 (socket says 172.21.15.156:35482) 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 -- 172.21.15.156:0/4094323854 learned_addr learned my addr 172.21.15.156:0/4094323854 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 -- 172.21.15.156:0/4094323854 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080d2120 msgr2=0x7fa7081e9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:56.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080d2120 0x7fa7081e9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 -- 172.21.15.156:0/4094323854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa70815e920 msgr2=0x7fa7081e9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa70815e920 0x7fa7081e9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 -- 172.21.15.156:0/4094323854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7080ce1a0 con 0x7fa70806cb10 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa70e38c640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 0x7fa7080cc210 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa70400d5d0 tx=0x7fa70400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7040125f0 con 0x7fa70806cb10 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa704019070 con 0x7fa70806cb10 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7040129f0 con 0x7fa70806cb10 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7080cb730 con 0x7fa70806cb10 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.161+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7081592c0 con 0x7fa70806cb10 2024-03-29T04:09:56.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.163+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7080a2480 con 0x7fa70806cb10 2024-03-29T04:09:56.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.163+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa704012b90 con 0x7fa70806cb10 2024-03-29T04:09:56.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.163+0000 7fa6feffd640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:56.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.164+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fa70408baa0 con 0x7fa70806cb10 2024-03-29T04:09:56.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.164+0000 7fa70db8b640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:56.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.164+0000 7fa70db8b640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8036000 tx=0x7fa6f8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:56.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.168+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa70405a150 con 0x7fa70806cb10 2024-03-29T04:09:56.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:56.505+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa70804f9d0 con 0x7fa70806cb10 2024-03-29T04:09:57.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.038+0000 7fa6feffd640 1 -- 172.21.15.156:0/4094323854 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7fa704059a50 con 0x7fa70806cb10 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 msgr2=0x7fa6ec06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8036000 tx=0x7fa6f8030000 comp rx=0 tx=0).stop 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 msgr2=0x7fa7080cc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 0x7fa7080cc210 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa70400d5d0 tx=0x7fa70400da90 comp rx=0 tx=0).stop 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 shutdown_connections 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa6ec06c560 0x7fa6ec06ea20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa70815e920 0x7fa7081e9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa7080d2120 0x7fa7081e9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 --2- 172.21.15.156:0/4094323854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa70806cb10 0x7fa7080cc210 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 >> 172.21.15.156:0/4094323854 conn(0x7fa70809d7a0 msgr2=0x7fa7080b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:57.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 shutdown_connections 2024-03-29T04:09:57.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.040+0000 7fa7105f0640 1 -- 172.21.15.156:0/4094323854 wait complete. 2024-03-29T04:09:57.031 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T04:09:57.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.157+0000 7fdbe4515640 1 Processor -- start 2024-03-29T04:09:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.157+0000 7fdbe4515640 1 -- start start 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe4515640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdbdc15e730 0x7fdbdc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe4515640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc0d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe4515640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdbdc0d4800 0x7fdbdc0d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe4515640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdbdc06c720 con 0x7fdbdc0d4800 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe4515640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdbdc0d4c90 con 0x7fdbdc15e730 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe4515640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdbdc0cfa20 con 0x7fdbdc0cf3e0 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdbdc0d4800 0x7fdbdc0d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdbdc0d4800 0x7fdbdc0d0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35494/0 (socket says 172.21.15.156:35494) 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 -- 172.21.15.156:0/1699827736 learned_addr learned my addr 172.21.15.156:0/1699827736 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 -- 172.21.15.156:0/1699827736 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 msgr2=0x7fdbdc0d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe1ab0640 1 --2- 172.21.15.156:0/1699827736 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc0d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe22b1640 1 --2- 172.21.15.156:0/1699827736 >> v2:172.21.15.203:3300/0 conn(0x7fdbdc15e730 0x7fdbdc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 --2- 172.21.15.156:0/1699827736 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc0d42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 -- 172.21.15.156:0/1699827736 >> v2:172.21.15.203:3300/0 conn(0x7fdbdc15e730 msgr2=0x7fdbdc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:57.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 --2- 172.21.15.156:0/1699827736 >> v2:172.21.15.203:3300/0 conn(0x7fdbdc15e730 0x7fdbdc0ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.158+0000 7fdbe2ab2640 1 -- 172.21.15.156:0/1699827736 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbdc0cfd00 con 0x7fdbdc0d4800 2024-03-29T04:09:57.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.159+0000 7fdbe2ab2640 1 --2- 172.21.15.156:0/1699827736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc0d0c60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc0d1f20 tx=0x7fdbd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6958cd7364cf3231 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.159+0000 7fdbe12af640 1 -- 172.21.15.156:0/1699827736 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd402ca70 con 0x7fdbdc0d4800 2024-03-29T04:09:57.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.159+0000 7fdbe12af640 1 -- 172.21.15.156:0/1699827736 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd402cc10 con 0x7fdbdc0d4800 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.159+0000 7fdbe12af640 1 -- 172.21.15.156:0/1699827736 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40343f0 con 0x7fdbdc0d4800 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 -- 172.21.15.156:0/1699827736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 msgr2=0x7fdbdc0d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 --2- 172.21.15.156:0/1699827736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc0d0c60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc0d1f20 tx=0x7fdbd402bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 -- 172.21.15.156:0/1699827736 shutdown_connections 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 --2- 172.21.15.156:0/1699827736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc0d0c60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 --2- 172.21.15.156:0/1699827736 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc0d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 --2- 172.21.15.156:0/1699827736 >> v2:172.21.15.203:3300/0 conn(0x7fdbdc15e730 0x7fdbdc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 -- 172.21.15.156:0/1699827736 >> 172.21.15.156:0/1699827736 conn(0x7fdbdc08bfd0 msgr2=0x7fdbdc09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 -- 172.21.15.156:0/1699827736 shutdown_connections 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.160+0000 7fdbe4515640 1 -- 172.21.15.156:0/1699827736 wait complete. 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.161+0000 7fdbe4515640 1 Processor -- start 2024-03-29T04:09:57.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.161+0000 7fdbe4515640 1 -- start start 2024-03-29T04:09:57.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc1f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbdc15e730 0x7fdbdc1f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc0d2220 con 0x7fdbdc0d4800 2024-03-29T04:09:57.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc0d02c0 con 0x7fdbdc15e730 2024-03-29T04:09:57.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdbdc15eb10 con 0x7fdbdc0cf3e0 2024-03-29T04:09:57.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc1f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc1f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35510/0 (socket says 172.21.15.156:35510) 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 -- 172.21.15.156:0/3918460462 learned_addr learned my addr 172.21.15.156:0/3918460462 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe2ab2640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbdc15e730 0x7fdbdc1f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe22b1640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 -- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 msgr2=0x7fdbdc147090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc147090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 -- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbdc15e730 msgr2=0x7fdbdc1f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbdc15e730 0x7fdbdc1f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 -- 172.21.15.156:0/3918460462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbdc1475d0 con 0x7fdbdc0d4800 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe2ab2640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbdc15e730 0x7fdbdc1f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:57.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe22b1640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:57.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe1ab0640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc1f7700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c7f0 tx=0x7fdbcc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc015070 con 0x7fdbdc0d4800 2024-03-29T04:09:57.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbcc004220 con 0x7fdbdc0d4800 2024-03-29T04:09:57.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbdc15a080 con 0x7fdbdc0d4800 2024-03-29T04:09:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc007620 con 0x7fdbdc0d4800 2024-03-29T04:09:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.162+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbdc15a640 con 0x7fdbdc0d4800 2024-03-29T04:09:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.163+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbdc068610 con 0x7fdbdc0d4800 2024-03-29T04:09:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.164+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdbcc0079d0 con 0x7fdbdc0d4800 2024-03-29T04:09:57.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.164+0000 7fdbd2ffd640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdbc006c530 0x7fdbc006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.165+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fdbcc08b970 con 0x7fdbdc0d4800 2024-03-29T04:09:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.165+0000 7fdbe22b1640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdbc006c530 0x7fdbc006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.165+0000 7fdbe22b1640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdbc006c530 0x7fdbc006e9f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b840 tx=0x7fdbd8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.173+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdbdc1475d0 con 0x7fdbdc0d4800 2024-03-29T04:09:57.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.391+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbdc06e270 con 0x7fdbdc0d4800 2024-03-29T04:09:57.371 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.392+0000 7fdbd2ffd640 1 -- 172.21.15.156:0/3918460462 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5190 (secure 0 0 0) 0x7fdbcc05a020 con 0x7fdbdc0d4800 2024-03-29T04:09:57.371 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:57.371 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:09:53.360769+0000","modified":"2024-03-29T04:09:56.506914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5310":{"gid":5310,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/788767406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":788767406},{"type":"v1","addr":"172.21.15.156:6837","nonce":788767406}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5310,"qdb_cluster":[5310]},"id":2}]} 2024-03-29T04:09:57.372 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 15 2024-03-29T04:09:57.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.394+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdbc006c530 msgr2=0x7fdbc006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdbc006c530 0x7fdbc006e9f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b840 tx=0x7fdbd8008040 comp rx=0 tx=0).stop 2024-03-29T04:09:57.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 msgr2=0x7fdbdc1f7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc1f7700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c7f0 tx=0x7fdbcc00ccb0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 shutdown_connections 2024-03-29T04:09:57.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdbc006c530 0x7fdbc006e9f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbdc15e730 0x7fdbdc1f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdbdc0d4800 0x7fdbdc1f7700 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 --2- 172.21.15.156:0/3918460462 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbdc0cf3e0 0x7fdbdc147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 >> 172.21.15.156:0/3918460462 conn(0x7fdbdc08bfd0 msgr2=0x7fdbdc15dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:57.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 shutdown_connections 2024-03-29T04:09:57.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.395+0000 7fdbe4515640 1 -- 172.21.15.156:0/3918460462 wait complete. 2024-03-29T04:09:57.385 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:09:57.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.508+0000 7fec98463640 1 Processor -- start 2024-03-29T04:09:57.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.508+0000 7fec98463640 1 -- start start 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec98463640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fec900d42c0 0x7fec900d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec98463640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fec9006cbb0 0x7fec9006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec98463640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec900d4c80 con 0x7fec9015e730 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec98463640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fec9006c720 con 0x7fec900d42c0 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec98463640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fec900cfa20 con 0x7fec9006cbb0 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec959fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fec900d42c0 0x7fec900d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec961ff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fec9015e730 0x7fec900ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44680/0 (socket says 172.21.15.156:44680) 2024-03-29T04:09:57.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec961ff640 1 -- 172.21.15.156:0/3138965690 learned_addr learned my addr 172.21.15.156:0/3138965690 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:57.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec96a00640 1 --2- 172.21.15.156:0/3138965690 >> v2:172.21.15.203:3300/0 conn(0x7fec9006cbb0 0x7fec9006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec959fe640 1 -- 172.21.15.156:0/3138965690 >> v2:172.21.15.203:3300/0 conn(0x7fec9006cbb0 msgr2=0x7fec9006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:57.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec959fe640 1 --2- 172.21.15.156:0/3138965690 >> v2:172.21.15.203:3300/0 conn(0x7fec9006cbb0 0x7fec9006cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec959fe640 1 -- 172.21.15.156:0/3138965690 >> v1:172.21.15.156:6789/0 conn(0x7fec9015e730 legacy=0x7fec900ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec959fe640 1 -- 172.21.15.156:0/3138965690 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec900cfd00 con 0x7fec900d42c0 2024-03-29T04:09:57.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.510+0000 7fec951fd640 1 -- 172.21.15.156:0/3138965690 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 770620865 0 0) 0x7fec900d4c80 con 0x7fec9015e730 2024-03-29T04:09:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.511+0000 7fec959fe640 1 --2- 172.21.15.156:0/3138965690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900d4740 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fec900cea10 tx=0x7fec8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69fb6a22c9fe0a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.511+0000 7fec951fd640 1 -- 172.21.15.156:0/3138965690 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84035020 con 0x7fec900d42c0 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.511+0000 7fec951fd640 1 -- 172.21.15.156:0/3138965690 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec84031830 con 0x7fec900d42c0 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.511+0000 7fec951fd640 1 -- 172.21.15.156:0/3138965690 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84031c00 con 0x7fec900d42c0 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 -- 172.21.15.156:0/3138965690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 msgr2=0x7fec900d4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 --2- 172.21.15.156:0/3138965690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900d4740 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fec900cea10 tx=0x7fec8402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 -- 172.21.15.156:0/3138965690 shutdown_connections 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 --2- 172.21.15.156:0/3138965690 >> v2:172.21.15.203:3300/0 conn(0x7fec9006cbb0 0x7fec9006cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 --2- 172.21.15.156:0/3138965690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900d4740 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 -- 172.21.15.156:0/3138965690 >> 172.21.15.156:0/3138965690 conn(0x7fec9008bfd0 msgr2=0x7fec9009d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 -- 172.21.15.156:0/3138965690 shutdown_connections 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.512+0000 7fec98463640 1 -- 172.21.15.156:0/3138965690 wait complete. 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.513+0000 7fec98463640 1 Processor -- start 2024-03-29T04:09:57.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.513+0000 7fec98463640 1 -- start start 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec98463640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec9006cbb0 0x7fec901f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec98463640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec98463640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec9015e730 0x7fec900cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec98463640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fec900d2220 con 0x7fec900d42c0 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec98463640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec900d02c0 con 0x7fec9006cbb0 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec98463640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fec9015eb10 con 0x7fec9015e730 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35526/0 (socket says 172.21.15.156:35526) 2024-03-29T04:09:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 -- 172.21.15.156:0/563429993 learned_addr learned my addr 172.21.15.156:0/563429993 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec96a00640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec9015e730 0x7fec900cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec961ff640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec9006cbb0 0x7fec901f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 -- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec9015e730 msgr2=0x7fec900cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec9015e730 0x7fec900cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 -- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec9006cbb0 msgr2=0x7fec901f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec9006cbb0 0x7fec901f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.514+0000 7fec959fe640 1 -- 172.21.15.156:0/563429993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec901f4870 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.515+0000 7fec959fe640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900ccaf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fec8402bee0 tx=0x7fec840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.515+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84041070 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.515+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8403f070 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.515+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec901f4b50 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.515+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84035040 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.515+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec901f5110 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.516+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90068610 con 0x7fec900d42c0 2024-03-29T04:09:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.517+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fec84038020 con 0x7fec900d42c0 2024-03-29T04:09:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.517+0000 7fec82ffd640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fec7406c560 0x7fec7406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.517+0000 7fec961ff640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fec7406c560 0x7fec7406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.517+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fec840acc20 con 0x7fec900d42c0 2024-03-29T04:09:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.518+0000 7fec961ff640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fec7406c560 0x7fec7406ea20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fec8c000f60 tx=0x7fec8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.522+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fec8407b2d0 con 0x7fec900d42c0 2024-03-29T04:09:57.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.704+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec900b2880 con 0x7fec900d42c0 2024-03-29T04:09:57.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.705+0000 7fec82ffd640 1 -- 172.21.15.156:0/563429993 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7fec8407abd0 con 0x7fec900d42c0 2024-03-29T04:09:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:57.685 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":34,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:55.015332+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:09:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fec7406c560 msgr2=0x7fec7406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fec7406c560 0x7fec7406ea20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fec8c000f60 tx=0x7fec8c009040 comp rx=0 tx=0).stop 2024-03-29T04:09:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 msgr2=0x7fec900ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900ccaf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fec8402bee0 tx=0x7fec840046c0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 shutdown_connections 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fec7406c560 0x7fec7406ea20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec9015e730 0x7fec900cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec900d42c0 0x7fec900ccaf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 --2- 172.21.15.156:0/563429993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec9006cbb0 0x7fec901f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 >> 172.21.15.156:0/563429993 conn(0x7fec9008bfd0 msgr2=0x7fec900b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 shutdown_connections 2024-03-29T04:09:57.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.708+0000 7fec98463640 1 -- 172.21.15.156:0/563429993 wait complete. 2024-03-29T04:09:57.698 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T04:09:57.699 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:09:57.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.820+0000 7f9f1ad8d640 1 Processor -- start 2024-03-29T04:09:57.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f1ad8d640 1 -- start start 2024-03-29T04:09:57.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f1ad8d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9f1415e730 0x7f9f140ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f1ad8d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f140d49d0 0x7f9f1406cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f1ad8d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f140cfa20 con 0x7f9f140cf3e0 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f1ad8d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9f1406c720 con 0x7f9f1415e730 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f1ad8d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9f140cfd00 con 0x7f9f140d49d0 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f13fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9f140cf3e0 0x7f9f140d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44682/0 (socket says 172.21.15.156:44682) 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f13fff640 1 -- 172.21.15.156:0/4019023456 learned_addr learned my addr 172.21.15.156:0/4019023456 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.821+0000 7f9f18b29640 1 --2- 172.21.15.156:0/4019023456 >> v2:172.21.15.203:3300/0 conn(0x7f9f1415e730 0x7f9f140ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f1932a640 1 --2- 172.21.15.156:0/4019023456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f140d49d0 0x7f9f1406cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f137fe640 1 -- 172.21.15.156:0/4019023456 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901318941 0 0) 0x7f9f140cfa20 con 0x7f9f140cf3e0 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f137fe640 1 -- 172.21.15.156:0/4019023456 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f04003520 con 0x7f9f140cf3e0 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f1932a640 1 -- 172.21.15.156:0/4019023456 >> v2:172.21.15.203:3300/0 conn(0x7f9f1415e730 msgr2=0x7f9f140ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f1932a640 1 --2- 172.21.15.156:0/4019023456 >> v2:172.21.15.203:3300/0 conn(0x7f9f1415e730 0x7f9f140ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f1932a640 1 -- 172.21.15.156:0/4019023456 >> v1:172.21.15.156:6789/0 conn(0x7f9f140cf3e0 legacy=0x7f9f140d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f1932a640 1 -- 172.21.15.156:0/4019023456 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f140cffe0 con 0x7f9f140d49d0 2024-03-29T04:09:57.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.822+0000 7f9f18b29640 1 --2- 172.21.15.156:0/4019023456 >> v2:172.21.15.203:3300/0 conn(0x7f9f1415e730 0x7f9f140ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:57.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1932a640 1 --2- 172.21.15.156:0/4019023456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f140d49d0 0x7f9f1406cbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009d90 tx=0x7f9f0c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88d217b08edc8268 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f137fe640 1 -- 172.21.15.156:0/4019023456 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c039070 con 0x7f9f140d49d0 2024-03-29T04:09:57.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f137fe640 1 -- 172.21.15.156:0/4019023456 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0c02cc50 con 0x7f9f140d49d0 2024-03-29T04:09:57.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f137fe640 1 -- 172.21.15.156:0/4019023456 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c034440 con 0x7f9f140d49d0 2024-03-29T04:09:57.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/4019023456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f140d49d0 msgr2=0x7f9f1406cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/4019023456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f140d49d0 0x7f9f1406cbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009d90 tx=0x7f9f0c02bea0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/4019023456 shutdown_connections 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/4019023456 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f140d49d0 0x7f9f1406cbb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/4019023456 >> v2:172.21.15.203:3300/0 conn(0x7f9f1415e730 0x7f9f140ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/4019023456 >> 172.21.15.156:0/4019023456 conn(0x7f9f1408bfd0 msgr2=0x7f9f1409d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/4019023456 shutdown_connections 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.823+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/4019023456 wait complete. 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 Processor -- start 2024-03-29T04:09:57.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 -- start start 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 0x7f9f140c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f140d49d0 0x7f9f14153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f1415e730 0x7f9f1415c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1415eb10 con 0x7f9f140cf3e0 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f140d2220 con 0x7f9f140d49d0 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.825+0000 7f9f1ad8d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1415ec90 con 0x7f9f1415e730 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 0x7f9f140c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 0x7f9f140c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35536/0 (socket says 172.21.15.156:35536) 2024-03-29T04:09:57.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 -- 172.21.15.156:0/243653454 learned_addr learned my addr 172.21.15.156:0/243653454 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f1932a640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f1415e730 0x7f9f1415c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f13fff640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f140d49d0 0x7f9f14153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 -- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f1415e730 msgr2=0x7f9f1415c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f1415e730 0x7f9f1415c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 -- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f140d49d0 msgr2=0x7f9f14153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f140d49d0 0x7f9f14153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 -- 172.21.15.156:0/243653454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f140c7ec0 con 0x7f9f140cf3e0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f18b29640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 0x7f9f140c8c80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800e990 tx=0x7f9f0800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08017070 con 0x7f9f140cf3e0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0800c9e0 con 0x7f9f140cf3e0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f140c81a0 con 0x7f9f140cf3e0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0800cdb0 con 0x7f9f140cf3e0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.826+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f141616c0 con 0x7f9f140cf3e0 2024-03-29T04:09:57.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.827+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f14068610 con 0x7f9f140cf3e0 2024-03-29T04:09:57.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.828+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9f0801a070 con 0x7f9f140cf3e0 2024-03-29T04:09:57.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.829+0000 7f9f117fa640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ef806c530 0x7f9ef806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:57.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.829+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f9f0808b910 con 0x7f9f140cf3e0 2024-03-29T04:09:57.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.829+0000 7f9f13fff640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ef806c530 0x7f9ef806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:57.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.829+0000 7f9f13fff640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ef806c530 0x7f9ef806e9f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9f14160440 tx=0x7f9f00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:57.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:57.833+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9f08059860 con 0x7f9f140cf3e0 2024-03-29T04:09:57.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.012+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f140b29f0 con 0x7f9ef806c530 2024-03-29T04:09:57.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.018+0000 7f9f117fa640 1 -- 172.21.15.156:0/243653454 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f9f140b29f0 con 0x7f9ef806c530 2024-03-29T04:09:58.000 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:58.001 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ef806c530 msgr2=0x7f9ef806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ef806c530 0x7f9ef806e9f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9f14160440 tx=0x7f9f00009040 comp rx=0 tx=0).stop 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 msgr2=0x7f9f140c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 0x7f9f140c8c80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800e990 tx=0x7f9f0800ee50 comp rx=0 tx=0).stop 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 shutdown_connections 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9ef806c530 0x7f9ef806e9f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f1415e730 0x7f9f1415c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f140d49d0 0x7f9f14153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 --2- 172.21.15.156:0/243653454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f140cf3e0 0x7f9f140c8c80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 >> 172.21.15.156:0/243653454 conn(0x7f9f1408bfd0 msgr2=0x7f9f140b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:58.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 shutdown_connections 2024-03-29T04:09:58.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.021+0000 7f9f1ad8d640 1 -- 172.21.15.156:0/243653454 wait complete. 2024-03-29T04:09:58.014 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-29T04:09:57.648583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237004,"kb_used_data":4188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737716,"statfs":{"total":773094113280,"available":772851421184,"internally_reserved":0,"allocated":4288512,"data_stored":2380528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.646651"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039499+0000","last_change":"2024-03-29T04:09:54.037102+0000","last_active":"2024-03-29T04:09:55.039499+0000","last_peered":"2024-03-29T04:09:55.039499+0000","last_clean":"2024-03-29T04:09:55.039499+0000","last_became_active":"2024-03-29T04:09:54.036788+0000","last_became_peered":"2024-03-29T04:09:54.036788+0000","last_unstale":"2024-03-29T04:09:55.039499+0000","last_undegraded":"2024-03-29T04:09:55.039499+0000","last_fullsized":"2024-03-29T04:09:55.039499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:24.718335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019400+0000","last_change":"2024-03-29T04:09:54.034758+0000","last_active":"2024-03-29T04:09:55.019400+0000","last_peered":"2024-03-29T04:09:55.019400+0000","last_clean":"2024-03-29T04:09:55.019400+0000","last_became_active":"2024-03-29T04:09:54.034482+0000","last_became_peered":"2024-03-29T04:09:54.034482+0000","last_unstale":"2024-03-29T04:09:55.019400+0000","last_undegraded":"2024-03-29T04:09:55.019400+0000","last_fullsized":"2024-03-29T04:09:55.019400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:48:24.105174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540258+0000","last_change":"2024-03-29T04:09:54.036045+0000","last_active":"2024-03-29T04:09:55.540258+0000","last_peered":"2024-03-29T04:09:55.540258+0000","last_clean":"2024-03-29T04:09:55.540258+0000","last_became_active":"2024-03-29T04:09:54.033549+0000","last_became_peered":"2024-03-29T04:09:54.033549+0000","last_unstale":"2024-03-29T04:09:55.540258+0000","last_undegraded":"2024-03-29T04:09:55.540258+0000","last_fullsized":"2024-03-29T04:09:55.540258+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:52.105155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040156+0000","last_change":"2024-03-29T04:09:54.031752+0000","last_active":"2024-03-29T04:09:55.040156+0000","last_peered":"2024-03-29T04:09:55.040156+0000","last_clean":"2024-03-29T04:09:55.040156+0000","last_became_active":"2024-03-29T04:09:54.031436+0000","last_became_peered":"2024-03-29T04:09:54.031436+0000","last_unstale":"2024-03-29T04:09:55.040156+0000","last_undegraded":"2024-03-29T04:09:55.040156+0000","last_fullsized":"2024-03-29T04:09:55.040156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:08.634564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540100+0000","last_change":"2024-03-29T04:09:54.036134+0000","last_active":"2024-03-29T04:09:55.540100+0000","last_peered":"2024-03-29T04:09:55.540100+0000","last_clean":"2024-03-29T04:09:55.540100+0000","last_became_active":"2024-03-29T04:09:54.035834+0000","last_became_peered":"2024-03-29T04:09:54.035834+0000","last_unstale":"2024-03-29T04:09:55.540100+0000","last_undegraded":"2024-03-29T04:09:55.540100+0000","last_fullsized":"2024-03-29T04:09:55.540100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.404392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018813+0000","last_change":"2024-03-29T04:09:54.038418+0000","last_active":"2024-03-29T04:09:55.018813+0000","last_peered":"2024-03-29T04:09:55.018813+0000","last_clean":"2024-03-29T04:09:55.018813+0000","last_became_active":"2024-03-29T04:09:54.038170+0000","last_became_peered":"2024-03-29T04:09:54.038170+0000","last_unstale":"2024-03-29T04:09:55.018813+0000","last_undegraded":"2024-03-29T04:09:55.018813+0000","last_fullsized":"2024-03-29T04:09:55.018813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:52.196258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022075+0000","last_change":"2024-03-29T04:09:54.042972+0000","last_active":"2024-03-29T04:09:55.022075+0000","last_peered":"2024-03-29T04:09:55.022075+0000","last_clean":"2024-03-29T04:09:55.022075+0000","last_became_active":"2024-03-29T04:09:54.042701+0000","last_became_peered":"2024-03-29T04:09:54.042701+0000","last_unstale":"2024-03-29T04:09:55.022075+0000","last_undegraded":"2024-03-29T04:09:55.022075+0000","last_fullsized":"2024-03-29T04:09:55.022075+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:55:57.600054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022123+0000","last_change":"2024-03-29T04:09:54.039996+0000","last_active":"2024-03-29T04:09:55.022123+0000","last_peered":"2024-03-29T04:09:55.022123+0000","last_clean":"2024-03-29T04:09:55.022123+0000","last_became_active":"2024-03-29T04:09:54.039712+0000","last_became_peered":"2024-03-29T04:09:54.039712+0000","last_unstale":"2024-03-29T04:09:55.022123+0000","last_undegraded":"2024-03-29T04:09:55.022123+0000","last_fullsized":"2024-03-29T04:09:55.022123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:27.334351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023068+0000","last_change":"2024-03-29T04:09:54.033289+0000","last_active":"2024-03-29T04:09:55.023068+0000","last_peered":"2024-03-29T04:09:55.023068+0000","last_clean":"2024-03-29T04:09:55.023068+0000","last_became_active":"2024-03-29T04:09:54.032168+0000","last_became_peered":"2024-03-29T04:09:54.032168+0000","last_unstale":"2024-03-29T04:09:55.023068+0000","last_undegraded":"2024-03-29T04:09:55.023068+0000","last_fullsized":"2024-03-29T04:09:55.023068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.103608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040287+0000","last_change":"2024-03-29T04:09:54.034286+0000","last_active":"2024-03-29T04:09:55.040287+0000","last_peered":"2024-03-29T04:09:55.040287+0000","last_clean":"2024-03-29T04:09:55.040287+0000","last_became_active":"2024-03-29T04:09:54.034031+0000","last_became_peered":"2024-03-29T04:09:54.034031+0000","last_unstale":"2024-03-29T04:09:55.040287+0000","last_undegraded":"2024-03-29T04:09:55.040287+0000","last_fullsized":"2024-03-29T04:09:55.040287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:58.350300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022004+0000","last_change":"2024-03-29T04:09:54.038944+0000","last_active":"2024-03-29T04:09:55.022004+0000","last_peered":"2024-03-29T04:09:55.022004+0000","last_clean":"2024-03-29T04:09:55.022004+0000","last_became_active":"2024-03-29T04:09:54.038672+0000","last_became_peered":"2024-03-29T04:09:54.038672+0000","last_unstale":"2024-03-29T04:09:55.022004+0000","last_undegraded":"2024-03-29T04:09:55.022004+0000","last_fullsized":"2024-03-29T04:09:55.022004+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:23.490248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540233+0000","last_change":"2024-03-29T04:09:54.034476+0000","last_active":"2024-03-29T04:09:55.540233+0000","last_peered":"2024-03-29T04:09:55.540233+0000","last_clean":"2024-03-29T04:09:55.540233+0000","last_became_active":"2024-03-29T04:09:54.034202+0000","last_became_peered":"2024-03-29T04:09:54.034202+0000","last_unstale":"2024-03-29T04:09:55.540233+0000","last_undegraded":"2024-03-29T04:09:55.540233+0000","last_fullsized":"2024-03-29T04:09:55.540233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:27.334351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540124+0000","last_change":"2024-03-29T04:09:54.025791+0000","last_active":"2024-03-29T04:09:55.540124+0000","last_peered":"2024-03-29T04:09:55.540124+0000","last_clean":"2024-03-29T04:09:55.540124+0000","last_became_active":"2024-03-29T04:09:54.025521+0000","last_became_peered":"2024-03-29T04:09:54.025521+0000","last_unstale":"2024-03-29T04:09:55.540124+0000","last_undegraded":"2024-03-29T04:09:55.540124+0000","last_fullsized":"2024-03-29T04:09:55.540124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:22.735465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018942+0000","last_change":"2024-03-29T04:09:54.027287+0000","last_active":"2024-03-29T04:09:55.018942+0000","last_peered":"2024-03-29T04:09:55.018942+0000","last_clean":"2024-03-29T04:09:55.018942+0000","last_became_active":"2024-03-29T04:09:54.026998+0000","last_became_peered":"2024-03-29T04:09:54.026998+0000","last_unstale":"2024-03-29T04:09:55.018942+0000","last_undegraded":"2024-03-29T04:09:55.018942+0000","last_fullsized":"2024-03-29T04:09:55.018942+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:58.930112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540066+0000","last_change":"2024-03-29T04:09:54.032319+0000","last_active":"2024-03-29T04:09:55.540066+0000","last_peered":"2024-03-29T04:09:55.540066+0000","last_clean":"2024-03-29T04:09:55.540066+0000","last_became_active":"2024-03-29T04:09:54.031957+0000","last_became_peered":"2024-03-29T04:09:54.031957+0000","last_unstale":"2024-03-29T04:09:55.540066+0000","last_undegraded":"2024-03-29T04:09:55.540066+0000","last_fullsized":"2024-03-29T04:09:55.540066+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:52.196258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022617+0000","last_change":"2024-03-29T04:09:54.044192+0000","last_active":"2024-03-29T04:09:55.022617+0000","last_peered":"2024-03-29T04:09:55.022617+0000","last_clean":"2024-03-29T04:09:55.022617+0000","last_became_active":"2024-03-29T04:09:54.043907+0000","last_became_peered":"2024-03-29T04:09:54.043907+0000","last_unstale":"2024-03-29T04:09:55.022617+0000","last_undegraded":"2024-03-29T04:09:55.022617+0000","last_fullsized":"2024-03-29T04:09:55.022617+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:15:50.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022436+0000","last_change":"2024-03-29T04:09:54.043341+0000","last_active":"2024-03-29T04:09:55.022436+0000","last_peered":"2024-03-29T04:09:55.022436+0000","last_clean":"2024-03-29T04:09:55.022436+0000","last_became_active":"2024-03-29T04:09:54.042757+0000","last_became_peered":"2024-03-29T04:09:54.042757+0000","last_unstale":"2024-03-29T04:09:55.022436+0000","last_undegraded":"2024-03-29T04:09:55.022436+0000","last_fullsized":"2024-03-29T04:09:55.022436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:48:24.105174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019515+0000","last_change":"2024-03-29T04:09:54.035421+0000","last_active":"2024-03-29T04:09:55.019515+0000","last_peered":"2024-03-29T04:09:55.019515+0000","last_clean":"2024-03-29T04:09:55.019515+0000","last_became_active":"2024-03-29T04:09:54.035160+0000","last_became_peered":"2024-03-29T04:09:54.035160+0000","last_unstale":"2024-03-29T04:09:55.019515+0000","last_undegraded":"2024-03-29T04:09:55.019515+0000","last_fullsized":"2024-03-29T04:09:55.019515+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:15:50.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018754+0000","last_change":"2024-03-29T04:09:54.034314+0000","last_active":"2024-03-29T04:09:55.018754+0000","last_peered":"2024-03-29T04:09:55.018754+0000","last_clean":"2024-03-29T04:09:55.018754+0000","last_became_active":"2024-03-29T04:09:54.034059+0000","last_became_peered":"2024-03-29T04:09:54.034059+0000","last_unstale":"2024-03-29T04:09:55.018754+0000","last_undegraded":"2024-03-29T04:09:55.018754+0000","last_fullsized":"2024-03-29T04:09:55.018754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:55:57.600054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540282+0000","last_change":"2024-03-29T04:09:54.030152+0000","last_active":"2024-03-29T04:09:55.540282+0000","last_peered":"2024-03-29T04:09:55.540282+0000","last_clean":"2024-03-29T04:09:55.540282+0000","last_became_active":"2024-03-29T04:09:54.029878+0000","last_became_peered":"2024-03-29T04:09:54.029878+0000","last_unstale":"2024-03-29T04:09:55.540282+0000","last_undegraded":"2024-03-29T04:09:55.540282+0000","last_fullsized":"2024-03-29T04:09:55.540282+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:15.243062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022972+0000","last_change":"2024-03-29T04:09:54.026691+0000","last_active":"2024-03-29T04:09:55.022972+0000","last_peered":"2024-03-29T04:09:55.022972+0000","last_clean":"2024-03-29T04:09:55.022972+0000","last_became_active":"2024-03-29T04:09:54.026254+0000","last_became_peered":"2024-03-29T04:09:54.026254+0000","last_unstale":"2024-03-29T04:09:55.022972+0000","last_undegraded":"2024-03-29T04:09:55.022972+0000","last_fullsized":"2024-03-29T04:09:55.022972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:42.226804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023249+0000","last_change":"2024-03-29T04:09:54.028422+0000","last_active":"2024-03-29T04:09:55.023249+0000","last_peered":"2024-03-29T04:09:55.023249+0000","last_clean":"2024-03-29T04:09:55.023249+0000","last_became_active":"2024-03-29T04:09:54.028128+0000","last_became_peered":"2024-03-29T04:09:54.028128+0000","last_unstale":"2024-03-29T04:09:55.023249+0000","last_undegraded":"2024-03-29T04:09:55.023249+0000","last_fullsized":"2024-03-29T04:09:55.023249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:16:37.027808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038118+0000","last_change":"2024-03-29T04:09:54.042812+0000","last_active":"2024-03-29T04:09:55.038118+0000","last_peered":"2024-03-29T04:09:55.038118+0000","last_clean":"2024-03-29T04:09:55.038118+0000","last_became_active":"2024-03-29T04:09:54.036389+0000","last_became_peered":"2024-03-29T04:09:54.036389+0000","last_unstale":"2024-03-29T04:09:55.038118+0000","last_undegraded":"2024-03-29T04:09:55.038118+0000","last_fullsized":"2024-03-29T04:09:55.038118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:35.196134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.539987+0000","last_change":"2024-03-29T04:09:54.028381+0000","last_active":"2024-03-29T04:09:55.539987+0000","last_peered":"2024-03-29T04:09:55.539987+0000","last_clean":"2024-03-29T04:09:55.539987+0000","last_became_active":"2024-03-29T04:09:54.028071+0000","last_became_peered":"2024-03-29T04:09:54.028071+0000","last_unstale":"2024-03-29T04:09:55.539987+0000","last_undegraded":"2024-03-29T04:09:55.539987+0000","last_fullsized":"2024-03-29T04:09:55.539987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:55:57.600054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040231+0000","last_change":"2024-03-29T04:09:54.038381+0000","last_active":"2024-03-29T04:09:55.040231+0000","last_peered":"2024-03-29T04:09:55.040231+0000","last_clean":"2024-03-29T04:09:55.040231+0000","last_became_active":"2024-03-29T04:09:54.038100+0000","last_became_peered":"2024-03-29T04:09:54.038100+0000","last_unstale":"2024-03-29T04:09:55.040231+0000","last_undegraded":"2024-03-29T04:09:55.040231+0000","last_fullsized":"2024-03-29T04:09:55.040231+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:13.319250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019602+0000","last_change":"2024-03-29T04:09:54.039126+0000","last_active":"2024-03-29T04:09:55.019602+0000","last_peered":"2024-03-29T04:09:55.019602+0000","last_clean":"2024-03-29T04:09:55.019602+0000","last_became_active":"2024-03-29T04:09:54.038872+0000","last_became_peered":"2024-03-29T04:09:54.038872+0000","last_unstale":"2024-03-29T04:09:55.019602+0000","last_undegraded":"2024-03-29T04:09:55.019602+0000","last_fullsized":"2024-03-29T04:09:55.019602+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:53:18.902650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040258+0000","last_change":"2024-03-29T04:09:54.031407+0000","last_active":"2024-03-29T04:09:55.040258+0000","last_peered":"2024-03-29T04:09:55.040258+0000","last_clean":"2024-03-29T04:09:55.040258+0000","last_became_active":"2024-03-29T04:09:54.031115+0000","last_became_peered":"2024-03-29T04:09:54.031115+0000","last_unstale":"2024-03-29T04:09:55.040258+0000","last_undegraded":"2024-03-29T04:09:55.040258+0000","last_fullsized":"2024-03-29T04:09:55.040258+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:32.934535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540348+0000","last_change":"2024-03-29T04:09:54.027543+0000","last_active":"2024-03-29T04:09:55.540348+0000","last_peered":"2024-03-29T04:09:55.540348+0000","last_clean":"2024-03-29T04:09:55.540348+0000","last_became_active":"2024-03-29T04:09:54.027208+0000","last_became_peered":"2024-03-29T04:09:54.027208+0000","last_unstale":"2024-03-29T04:09:55.540348+0000","last_undegraded":"2024-03-29T04:09:55.540348+0000","last_fullsized":"2024-03-29T04:09:55.540348+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:48:24.105174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023342+0000","last_change":"2024-03-29T04:09:54.038213+0000","last_active":"2024-03-29T04:09:55.023342+0000","last_peered":"2024-03-29T04:09:55.023342+0000","last_clean":"2024-03-29T04:09:55.023342+0000","last_became_active":"2024-03-29T04:09:54.037941+0000","last_became_peered":"2024-03-29T04:09:54.037941+0000","last_unstale":"2024-03-29T04:09:55.023342+0000","last_undegraded":"2024-03-29T04:09:55.023342+0000","last_fullsized":"2024-03-29T04:09:55.023342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:45.126080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.021846+0000","last_change":"2024-03-29T04:09:54.042301+0000","last_active":"2024-03-29T04:09:55.021846+0000","last_peered":"2024-03-29T04:09:55.021846+0000","last_clean":"2024-03-29T04:09:55.021846+0000","last_became_active":"2024-03-29T04:09:54.041984+0000","last_became_peered":"2024-03-29T04:09:54.041984+0000","last_unstale":"2024-03-29T04:09:55.021846+0000","last_undegraded":"2024-03-29T04:09:55.021846+0000","last_fullsized":"2024-03-29T04:09:55.021846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:58.350300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039318+0000","last_change":"2024-03-29T04:09:54.031350+0000","last_active":"2024-03-29T04:09:55.039318+0000","last_peered":"2024-03-29T04:09:55.039318+0000","last_clean":"2024-03-29T04:09:55.039318+0000","last_became_active":"2024-03-29T04:09:54.030955+0000","last_became_peered":"2024-03-29T04:09:54.030955+0000","last_unstale":"2024-03-29T04:09:55.039318+0000","last_undegraded":"2024-03-29T04:09:55.039318+0000","last_fullsized":"2024-03-29T04:09:55.039318+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:47.366680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022466+0000","last_change":"2024-03-29T04:09:54.036526+0000","last_active":"2024-03-29T04:09:55.022466+0000","last_peered":"2024-03-29T04:09:55.022466+0000","last_clean":"2024-03-29T04:09:55.022466+0000","last_became_active":"2024-03-29T04:09:54.036185+0000","last_became_peered":"2024-03-29T04:09:54.036185+0000","last_unstale":"2024-03-29T04:09:55.022466+0000","last_undegraded":"2024-03-29T04:09:55.022466+0000","last_fullsized":"2024-03-29T04:09:55.022466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:16:21.172007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038018+0000","last_change":"2024-03-29T04:09:54.035509+0000","last_active":"2024-03-29T04:09:55.038018+0000","last_peered":"2024-03-29T04:09:55.038018+0000","last_clean":"2024-03-29T04:09:55.038018+0000","last_became_active":"2024-03-29T04:09:54.035214+0000","last_became_peered":"2024-03-29T04:09:54.035214+0000","last_unstale":"2024-03-29T04:09:55.038018+0000","last_undegraded":"2024-03-29T04:09:55.038018+0000","last_fullsized":"2024-03-29T04:09:55.038018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:42.226804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039560+0000","last_change":"2024-03-29T04:09:53.028108+0000","last_active":"2024-03-29T04:09:55.039560+0000","last_peered":"2024-03-29T04:09:55.039560+0000","last_clean":"2024-03-29T04:09:55.039560+0000","last_became_active":"2024-03-29T04:09:53.027794+0000","last_became_peered":"2024-03-29T04:09:53.027794+0000","last_unstale":"2024-03-29T04:09:55.039560+0000","last_undegraded":"2024-03-29T04:09:55.039560+0000","last_fullsized":"2024-03-29T04:09:55.039560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:51.906050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039531+0000","last_change":"2024-03-29T04:09:54.035684+0000","last_active":"2024-03-29T04:09:55.039531+0000","last_peered":"2024-03-29T04:09:55.039531+0000","last_clean":"2024-03-29T04:09:55.039531+0000","last_became_active":"2024-03-29T04:09:54.035407+0000","last_became_peered":"2024-03-29T04:09:54.035407+0000","last_unstale":"2024-03-29T04:09:55.039531+0000","last_undegraded":"2024-03-29T04:09:55.039531+0000","last_fullsized":"2024-03-29T04:09:55.039531+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:20.156066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.037941+0000","last_change":"2024-03-29T04:09:53.019049+0000","last_active":"2024-03-29T04:09:55.037941+0000","last_peered":"2024-03-29T04:09:55.037941+0000","last_clean":"2024-03-29T04:09:55.037941+0000","last_became_active":"2024-03-29T04:09:53.018782+0000","last_became_peered":"2024-03-29T04:09:53.018782+0000","last_unstale":"2024-03-29T04:09:55.037941+0000","last_undegraded":"2024-03-29T04:09:55.037941+0000","last_fullsized":"2024-03-29T04:09:55.037941+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:02.067736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023142+0000","last_change":"2024-03-29T04:09:54.041127+0000","last_active":"2024-03-29T04:09:55.023142+0000","last_peered":"2024-03-29T04:09:55.023142+0000","last_clean":"2024-03-29T04:09:55.023142+0000","last_became_active":"2024-03-29T04:09:54.040801+0000","last_became_peered":"2024-03-29T04:09:54.040801+0000","last_unstale":"2024-03-29T04:09:55.023142+0000","last_undegraded":"2024-03-29T04:09:55.023142+0000","last_fullsized":"2024-03-29T04:09:55.023142+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:51.065761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018846+0000","last_change":"2024-03-29T04:09:53.017697+0000","last_active":"2024-03-29T04:09:55.018846+0000","last_peered":"2024-03-29T04:09:55.018846+0000","last_clean":"2024-03-29T04:09:55.018846+0000","last_became_active":"2024-03-29T04:09:53.017395+0000","last_became_peered":"2024-03-29T04:09:53.017395+0000","last_unstale":"2024-03-29T04:09:55.018846+0000","last_undegraded":"2024-03-29T04:09:55.018846+0000","last_fullsized":"2024-03-29T04:09:55.018846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:23.714341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040296+0000","last_change":"2024-03-29T04:09:54.029204+0000","last_active":"2024-03-29T04:09:55.040296+0000","last_peered":"2024-03-29T04:09:55.040296+0000","last_clean":"2024-03-29T04:09:55.040296+0000","last_became_active":"2024-03-29T04:09:54.028909+0000","last_became_peered":"2024-03-29T04:09:54.028909+0000","last_unstale":"2024-03-29T04:09:55.040296+0000","last_undegraded":"2024-03-29T04:09:55.040296+0000","last_fullsized":"2024-03-29T04:09:55.040296+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:38.793033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540233+0000","last_change":"2024-03-29T04:09:53.023779+0000","last_active":"2024-03-29T04:09:55.540233+0000","last_peered":"2024-03-29T04:09:55.540233+0000","last_clean":"2024-03-29T04:09:55.540233+0000","last_became_active":"2024-03-29T04:09:53.023516+0000","last_became_peered":"2024-03-29T04:09:53.023516+0000","last_unstale":"2024-03-29T04:09:55.540233+0000","last_undegraded":"2024-03-29T04:09:55.540233+0000","last_fullsized":"2024-03-29T04:09:55.540233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:01.927119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022036+0000","last_change":"2024-03-29T04:09:54.036519+0000","last_active":"2024-03-29T04:09:55.022036+0000","last_peered":"2024-03-29T04:09:55.022036+0000","last_clean":"2024-03-29T04:09:55.022036+0000","last_became_active":"2024-03-29T04:09:54.036200+0000","last_became_peered":"2024-03-29T04:09:54.036200+0000","last_unstale":"2024-03-29T04:09:55.022036+0000","last_undegraded":"2024-03-29T04:09:55.022036+0000","last_fullsized":"2024-03-29T04:09:55.022036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:52.196258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023443+0000","last_change":"2024-03-29T04:09:53.023448+0000","last_active":"2024-03-29T04:09:55.023443+0000","last_peered":"2024-03-29T04:09:55.023443+0000","last_clean":"2024-03-29T04:09:55.023443+0000","last_became_active":"2024-03-29T04:09:53.023020+0000","last_became_peered":"2024-03-29T04:09:53.023020+0000","last_unstale":"2024-03-29T04:09:55.023443+0000","last_undegraded":"2024-03-29T04:09:55.023443+0000","last_fullsized":"2024-03-29T04:09:55.023443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:27.437037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540119+0000","last_change":"2024-03-29T04:09:54.027552+0000","last_active":"2024-03-29T04:09:55.540119+0000","last_peered":"2024-03-29T04:09:55.540119+0000","last_clean":"2024-03-29T04:09:55.540119+0000","last_became_active":"2024-03-29T04:09:54.027266+0000","last_became_peered":"2024-03-29T04:09:54.027266+0000","last_unstale":"2024-03-29T04:09:55.540119+0000","last_undegraded":"2024-03-29T04:09:55.540119+0000","last_fullsized":"2024-03-29T04:09:55.540119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:58.930112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022896+0000","last_change":"2024-03-29T04:09:53.019184+0000","last_active":"2024-03-29T04:09:55.022896+0000","last_peered":"2024-03-29T04:09:55.022896+0000","last_clean":"2024-03-29T04:09:55.022896+0000","last_became_active":"2024-03-29T04:09:53.018866+0000","last_became_peered":"2024-03-29T04:09:53.018866+0000","last_unstale":"2024-03-29T04:09:55.022896+0000","last_undegraded":"2024-03-29T04:09:55.022896+0000","last_fullsized":"2024-03-29T04:09:55.022896+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:17.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022533+0000","last_change":"2024-03-29T04:09:54.031799+0000","last_active":"2024-03-29T04:09:55.022533+0000","last_peered":"2024-03-29T04:09:55.022533+0000","last_clean":"2024-03-29T04:09:55.022533+0000","last_became_active":"2024-03-29T04:09:54.031530+0000","last_became_peered":"2024-03-29T04:09:54.031530+0000","last_unstale":"2024-03-29T04:09:55.022533+0000","last_undegraded":"2024-03-29T04:09:55.022533+0000","last_fullsized":"2024-03-29T04:09:55.022533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:20.156066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018657+0000","last_change":"2024-03-29T04:09:53.018030+0000","last_active":"2024-03-29T04:09:55.018657+0000","last_peered":"2024-03-29T04:09:55.018657+0000","last_clean":"2024-03-29T04:09:55.018657+0000","last_became_active":"2024-03-29T04:09:53.017329+0000","last_became_peered":"2024-03-29T04:09:53.017329+0000","last_unstale":"2024-03-29T04:09:55.018657+0000","last_undegraded":"2024-03-29T04:09:55.018657+0000","last_fullsized":"2024-03-29T04:09:55.018657+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:16:36.023814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039773+0000","last_change":"2024-03-29T04:09:54.031093+0000","last_active":"2024-03-29T04:09:55.039773+0000","last_peered":"2024-03-29T04:09:55.039773+0000","last_clean":"2024-03-29T04:09:55.039773+0000","last_became_active":"2024-03-29T04:09:54.030685+0000","last_became_peered":"2024-03-29T04:09:54.030685+0000","last_unstale":"2024-03-29T04:09:55.039773+0000","last_undegraded":"2024-03-29T04:09:55.039773+0000","last_fullsized":"2024-03-29T04:09:55.039773+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:45.126080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022658+0000","last_change":"2024-03-29T04:09:53.022641+0000","last_active":"2024-03-29T04:09:55.022658+0000","last_peered":"2024-03-29T04:09:55.022658+0000","last_clean":"2024-03-29T04:09:55.022658+0000","last_became_active":"2024-03-29T04:09:53.022338+0000","last_became_peered":"2024-03-29T04:09:53.022338+0000","last_unstale":"2024-03-29T04:09:55.022658+0000","last_undegraded":"2024-03-29T04:09:55.022658+0000","last_fullsized":"2024-03-29T04:09:55.022658+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:51.246947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540414+0000","last_change":"2024-03-29T04:09:54.030623+0000","last_active":"2024-03-29T04:09:55.540414+0000","last_peered":"2024-03-29T04:09:55.540414+0000","last_clean":"2024-03-29T04:09:55.540414+0000","last_became_active":"2024-03-29T04:09:54.030352+0000","last_became_peered":"2024-03-29T04:09:54.030352+0000","last_unstale":"2024-03-29T04:09:55.540414+0000","last_undegraded":"2024-03-29T04:09:55.540414+0000","last_fullsized":"2024-03-29T04:09:55.540414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:15:50.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019138+0000","last_change":"2024-03-29T04:09:53.024582+0000","last_active":"2024-03-29T04:09:55.019138+0000","last_peered":"2024-03-29T04:09:55.019138+0000","last_clean":"2024-03-29T04:09:55.019138+0000","last_became_active":"2024-03-29T04:09:53.024238+0000","last_became_peered":"2024-03-29T04:09:53.024238+0000","last_unstale":"2024-03-29T04:09:55.019138+0000","last_undegraded":"2024-03-29T04:09:55.019138+0000","last_fullsized":"2024-03-29T04:09:55.019138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:34.192140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019054+0000","last_change":"2024-03-29T04:09:54.036761+0000","last_active":"2024-03-29T04:09:55.019054+0000","last_peered":"2024-03-29T04:09:55.019054+0000","last_clean":"2024-03-29T04:09:55.019054+0000","last_became_active":"2024-03-29T04:09:54.036503+0000","last_became_peered":"2024-03-29T04:09:54.036503+0000","last_unstale":"2024-03-29T04:09:55.019054+0000","last_undegraded":"2024-03-29T04:09:55.019054+0000","last_fullsized":"2024-03-29T04:09:55.019054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:52.105155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038072+0000","last_change":"2024-03-29T04:09:53.016622+0000","last_active":"2024-03-29T04:09:55.038072+0000","last_peered":"2024-03-29T04:09:55.038072+0000","last_clean":"2024-03-29T04:09:55.038072+0000","last_became_active":"2024-03-29T04:09:53.016288+0000","last_became_peered":"2024-03-29T04:09:53.016288+0000","last_unstale":"2024-03-29T04:09:55.038072+0000","last_undegraded":"2024-03-29T04:09:55.038072+0000","last_fullsized":"2024-03-29T04:09:55.038072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:40:36.681703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018968+0000","last_change":"2024-03-29T04:09:54.034065+0000","last_active":"2024-03-29T04:09:55.018968+0000","last_peered":"2024-03-29T04:09:55.018968+0000","last_clean":"2024-03-29T04:09:55.018968+0000","last_became_active":"2024-03-29T04:09:54.033795+0000","last_became_peered":"2024-03-29T04:09:54.033795+0000","last_unstale":"2024-03-29T04:09:55.018968+0000","last_undegraded":"2024-03-29T04:09:55.018968+0000","last_fullsized":"2024-03-29T04:09:55.018968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:27.334351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540405+0000","last_change":"2024-03-29T04:09:53.019135+0000","last_active":"2024-03-29T04:09:55.540405+0000","last_peered":"2024-03-29T04:09:55.540405+0000","last_clean":"2024-03-29T04:09:55.540405+0000","last_became_active":"2024-03-29T04:09:53.018856+0000","last_became_peered":"2024-03-29T04:09:53.018856+0000","last_unstale":"2024-03-29T04:09:55.540405+0000","last_undegraded":"2024-03-29T04:09:55.540405+0000","last_fullsized":"2024-03-29T04:09:55.540405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:50.061767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540295+0000","last_change":"2024-03-29T04:09:54.034651+0000","last_active":"2024-03-29T04:09:55.540295+0000","last_peered":"2024-03-29T04:09:55.540295+0000","last_clean":"2024-03-29T04:09:55.540295+0000","last_became_active":"2024-03-29T04:09:54.033682+0000","last_became_peered":"2024-03-29T04:09:54.033682+0000","last_unstale":"2024-03-29T04:09:55.540295+0000","last_undegraded":"2024-03-29T04:09:55.540295+0000","last_fullsized":"2024-03-29T04:09:55.540295+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:16:21.172007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019050+0000","last_change":"2024-03-29T04:09:53.018264+0000","last_active":"2024-03-29T04:09:55.019050+0000","last_peered":"2024-03-29T04:09:55.019050+0000","last_clean":"2024-03-29T04:09:55.019050+0000","last_became_active":"2024-03-29T04:09:53.017056+0000","last_became_peered":"2024-03-29T04:09:53.017056+0000","last_unstale":"2024-03-29T04:09:55.019050+0000","last_undegraded":"2024-03-29T04:09:55.019050+0000","last_fullsized":"2024-03-29T04:09:55.019050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:19.152072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039798+0000","last_change":"2024-03-29T04:09:54.039934+0000","last_active":"2024-03-29T04:09:55.039798+0000","last_peered":"2024-03-29T04:09:55.039798+0000","last_clean":"2024-03-29T04:09:55.039798+0000","last_became_active":"2024-03-29T04:09:54.039669+0000","last_became_peered":"2024-03-29T04:09:54.039669+0000","last_unstale":"2024-03-29T04:09:55.039798+0000","last_undegraded":"2024-03-29T04:09:55.039798+0000","last_fullsized":"2024-03-29T04:09:55.039798+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:45:58.678448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023541+0000","last_change":"2024-03-29T04:09:53.027098+0000","last_active":"2024-03-29T04:09:55.023541+0000","last_peered":"2024-03-29T04:09:55.023541+0000","last_clean":"2024-03-29T04:09:55.023541+0000","last_became_active":"2024-03-29T04:09:53.026801+0000","last_became_peered":"2024-03-29T04:09:53.026801+0000","last_unstale":"2024-03-29T04:09:55.023541+0000","last_undegraded":"2024-03-29T04:09:55.023541+0000","last_fullsized":"2024-03-29T04:09:55.023541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:21.731470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019298+0000","last_change":"2024-03-29T04:09:54.033442+0000","last_active":"2024-03-29T04:09:55.019298+0000","last_peered":"2024-03-29T04:09:55.019298+0000","last_clean":"2024-03-29T04:09:55.019298+0000","last_became_active":"2024-03-29T04:09:54.033138+0000","last_became_peered":"2024-03-29T04:09:54.033138+0000","last_unstale":"2024-03-29T04:09:55.019298+0000","last_undegraded":"2024-03-29T04:09:55.019298+0000","last_fullsized":"2024-03-29T04:09:55.019298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:16:21.172007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039898+0000","last_change":"2024-03-29T04:09:53.025274+0000","last_active":"2024-03-29T04:09:55.039898+0000","last_peered":"2024-03-29T04:09:55.039898+0000","last_clean":"2024-03-29T04:09:55.039898+0000","last_became_active":"2024-03-29T04:09:53.025002+0000","last_became_peered":"2024-03-29T04:09:53.025002+0000","last_unstale":"2024-03-29T04:09:55.039898+0000","last_undegraded":"2024-03-29T04:09:55.039898+0000","last_fullsized":"2024-03-29T04:09:55.039898+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:02.067736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022269+0000","last_change":"2024-03-29T04:09:54.037254+0000","last_active":"2024-03-29T04:09:55.022269+0000","last_peered":"2024-03-29T04:09:55.022269+0000","last_clean":"2024-03-29T04:09:55.022269+0000","last_became_active":"2024-03-29T04:09:54.036970+0000","last_became_peered":"2024-03-29T04:09:54.036970+0000","last_unstale":"2024-03-29T04:09:55.022269+0000","last_undegraded":"2024-03-29T04:09:55.022269+0000","last_fullsized":"2024-03-29T04:09:55.022269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:52.105155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018598+0000","last_change":"2024-03-29T04:09:53.025284+0000","last_active":"2024-03-29T04:09:55.018598+0000","last_peered":"2024-03-29T04:09:55.018598+0000","last_clean":"2024-03-29T04:09:55.018598+0000","last_became_active":"2024-03-29T04:09:53.025018+0000","last_became_peered":"2024-03-29T04:09:53.025018+0000","last_unstale":"2024-03-29T04:09:55.018598+0000","last_undegraded":"2024-03-29T04:09:55.018598+0000","last_fullsized":"2024-03-29T04:09:55.018598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:06.969462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039442+0000","last_change":"2024-03-29T04:09:54.036284+0000","last_active":"2024-03-29T04:09:55.039442+0000","last_peered":"2024-03-29T04:09:55.039442+0000","last_clean":"2024-03-29T04:09:55.039442+0000","last_became_active":"2024-03-29T04:09:54.035804+0000","last_became_peered":"2024-03-29T04:09:54.035804+0000","last_unstale":"2024-03-29T04:09:55.039442+0000","last_undegraded":"2024-03-29T04:09:55.039442+0000","last_fullsized":"2024-03-29T04:09:55.039442+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:07.973456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018732+0000","last_change":"2024-03-29T04:09:53.016095+0000","last_active":"2024-03-29T04:09:55.018732+0000","last_peered":"2024-03-29T04:09:55.018732+0000","last_clean":"2024-03-29T04:09:55.018732+0000","last_became_active":"2024-03-29T04:09:53.015800+0000","last_became_peered":"2024-03-29T04:09:53.015800+0000","last_unstale":"2024-03-29T04:09:55.018732+0000","last_undegraded":"2024-03-29T04:09:55.018732+0000","last_fullsized":"2024-03-29T04:09:55.018732+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:23.099613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023061+0000","last_change":"2024-03-29T04:09:54.028344+0000","last_active":"2024-03-29T04:09:55.023061+0000","last_peered":"2024-03-29T04:09:55.023061+0000","last_clean":"2024-03-29T04:09:55.023061+0000","last_became_active":"2024-03-29T04:09:54.028063+0000","last_became_peered":"2024-03-29T04:09:54.028063+0000","last_unstale":"2024-03-29T04:09:55.023061+0000","last_undegraded":"2024-03-29T04:09:55.023061+0000","last_fullsized":"2024-03-29T04:09:55.023061+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:35.196134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040418+0000","last_change":"2024-03-29T04:09:53.021619+0000","last_active":"2024-03-29T04:09:55.040418+0000","last_peered":"2024-03-29T04:09:55.040418+0000","last_clean":"2024-03-29T04:09:55.040418+0000","last_became_active":"2024-03-29T04:09:53.021319+0000","last_became_peered":"2024-03-29T04:09:53.021319+0000","last_unstale":"2024-03-29T04:09:55.040418+0000","last_undegraded":"2024-03-29T04:09:55.040418+0000","last_fullsized":"2024-03-29T04:09:55.040418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:46.362685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540468+0000","last_change":"2024-03-29T04:09:54.034324+0000","last_active":"2024-03-29T04:09:55.540468+0000","last_peered":"2024-03-29T04:09:55.540468+0000","last_clean":"2024-03-29T04:09:55.540468+0000","last_became_active":"2024-03-29T04:09:54.033722+0000","last_became_peered":"2024-03-29T04:09:54.033722+0000","last_unstale":"2024-03-29T04:09:55.540468+0000","last_undegraded":"2024-03-29T04:09:55.540468+0000","last_fullsized":"2024-03-29T04:09:55.540468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:53:18.902650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022953+0000","last_change":"2024-03-29T04:09:53.027071+0000","last_active":"2024-03-29T04:09:55.022953+0000","last_peered":"2024-03-29T04:09:55.022953+0000","last_clean":"2024-03-29T04:09:55.022953+0000","last_became_active":"2024-03-29T04:09:53.026745+0000","last_became_peered":"2024-03-29T04:09:53.026745+0000","last_unstale":"2024-03-29T04:09:55.022953+0000","last_undegraded":"2024-03-29T04:09:55.022953+0000","last_fullsized":"2024-03-29T04:09:55.022953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:05.400397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040119+0000","last_change":"2024-03-29T04:09:54.029894+0000","last_active":"2024-03-29T04:09:55.040119+0000","last_peered":"2024-03-29T04:09:55.040119+0000","last_clean":"2024-03-29T04:09:55.040119+0000","last_became_active":"2024-03-29T04:09:54.029626+0000","last_became_peered":"2024-03-29T04:09:54.029626+0000","last_unstale":"2024-03-29T04:09:55.040119+0000","last_undegraded":"2024-03-29T04:09:55.040119+0000","last_fullsized":"2024-03-29T04:09:55.040119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:33:12.111409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023237+0000","last_change":"2024-03-29T04:09:53.017139+0000","last_active":"2024-03-29T04:09:55.023237+0000","last_peered":"2024-03-29T04:09:55.023237+0000","last_clean":"2024-03-29T04:09:55.023237+0000","last_became_active":"2024-03-29T04:09:53.016832+0000","last_became_peered":"2024-03-29T04:09:53.016832+0000","last_unstale":"2024-03-29T04:09:55.023237+0000","last_undegraded":"2024-03-29T04:09:55.023237+0000","last_fullsized":"2024-03-29T04:09:55.023237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:41.712929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540302+0000","last_change":"2024-03-29T04:09:54.036616+0000","last_active":"2024-03-29T04:09:55.540302+0000","last_peered":"2024-03-29T04:09:55.540302+0000","last_clean":"2024-03-29T04:09:55.540302+0000","last_became_active":"2024-03-29T04:09:54.036325+0000","last_became_peered":"2024-03-29T04:09:54.036325+0000","last_unstale":"2024-03-29T04:09:55.540302+0000","last_undegraded":"2024-03-29T04:09:55.540302+0000","last_fullsized":"2024-03-29T04:09:55.540302+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:49.771990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023265+0000","last_change":"2024-03-29T04:09:53.019125+0000","last_active":"2024-03-29T04:09:55.023265+0000","last_peered":"2024-03-29T04:09:55.023265+0000","last_clean":"2024-03-29T04:09:55.023265+0000","last_became_active":"2024-03-29T04:09:53.018823+0000","last_became_peered":"2024-03-29T04:09:53.018823+0000","last_unstale":"2024-03-29T04:09:55.023265+0000","last_undegraded":"2024-03-29T04:09:55.023265+0000","last_fullsized":"2024-03-29T04:09:55.023265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:42.552906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540093+0000","last_change":"2024-03-29T04:09:54.031341+0000","last_active":"2024-03-29T04:09:55.540093+0000","last_peered":"2024-03-29T04:09:55.540093+0000","last_clean":"2024-03-29T04:09:55.540093+0000","last_became_active":"2024-03-29T04:09:54.031071+0000","last_became_peered":"2024-03-29T04:09:54.031071+0000","last_unstale":"2024-03-29T04:09:55.540093+0000","last_undegraded":"2024-03-29T04:09:55.540093+0000","last_fullsized":"2024-03-29T04:09:55.540093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:52.250941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038105+0000","last_change":"2024-03-29T04:09:53.023196+0000","last_active":"2024-03-29T04:09:55.038105+0000","last_peered":"2024-03-29T04:09:55.038105+0000","last_clean":"2024-03-29T04:09:55.038105+0000","last_became_active":"2024-03-29T04:09:53.022902+0000","last_became_peered":"2024-03-29T04:09:53.022902+0000","last_unstale":"2024-03-29T04:09:55.038105+0000","last_undegraded":"2024-03-29T04:09:55.038105+0000","last_fullsized":"2024-03-29T04:09:55.038105+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.727836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540243+0000","last_change":"2024-03-29T04:09:54.023943+0000","last_active":"2024-03-29T04:09:55.540243+0000","last_peered":"2024-03-29T04:09:55.540243+0000","last_clean":"2024-03-29T04:09:55.540243+0000","last_became_active":"2024-03-29T04:09:54.023692+0000","last_became_peered":"2024-03-29T04:09:54.023692+0000","last_unstale":"2024-03-29T04:09:55.540243+0000","last_undegraded":"2024-03-29T04:09:55.540243+0000","last_fullsized":"2024-03-29T04:09:55.540243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:52.910044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039387+0000","last_change":"2024-03-29T04:09:53.027847+0000","last_active":"2024-03-29T04:09:55.039387+0000","last_peered":"2024-03-29T04:09:55.039387+0000","last_clean":"2024-03-29T04:09:55.039387+0000","last_became_active":"2024-03-29T04:09:53.027496+0000","last_became_peered":"2024-03-29T04:09:53.027496+0000","last_unstale":"2024-03-29T04:09:55.039387+0000","last_undegraded":"2024-03-29T04:09:55.039387+0000","last_fullsized":"2024-03-29T04:09:55.039387+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:21.731470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540108+0000","last_change":"2024-03-29T04:09:54.023583+0000","last_active":"2024-03-29T04:09:55.540108+0000","last_peered":"2024-03-29T04:09:55.540108+0000","last_clean":"2024-03-29T04:09:55.540108+0000","last_became_active":"2024-03-29T04:09:54.023273+0000","last_became_peered":"2024-03-29T04:09:54.023273+0000","last_unstale":"2024-03-29T04:09:55.540108+0000","last_undegraded":"2024-03-29T04:09:55.540108+0000","last_fullsized":"2024-03-29T04:09:55.540108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:28.441032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023033+0000","last_change":"2024-03-29T04:09:53.020050+0000","last_active":"2024-03-29T04:09:55.023033+0000","last_peered":"2024-03-29T04:09:55.023033+0000","last_clean":"2024-03-29T04:09:55.023033+0000","last_became_active":"2024-03-29T04:09:53.019803+0000","last_became_peered":"2024-03-29T04:09:53.019803+0000","last_unstale":"2024-03-29T04:09:55.023033+0000","last_undegraded":"2024-03-29T04:09:55.023033+0000","last_fullsized":"2024-03-29T04:09:55.023033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:51.906050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022077+0000","last_change":"2024-03-29T04:09:54.042188+0000","last_active":"2024-03-29T04:09:55.022077+0000","last_peered":"2024-03-29T04:09:55.022077+0000","last_clean":"2024-03-29T04:09:55.022077+0000","last_became_active":"2024-03-29T04:09:54.041896+0000","last_became_peered":"2024-03-29T04:09:54.041896+0000","last_unstale":"2024-03-29T04:09:55.022077+0000","last_undegraded":"2024-03-29T04:09:55.022077+0000","last_fullsized":"2024-03-29T04:09:55.022077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:58.930112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.539918+0000","last_change":"2024-03-29T04:09:53.022415+0000","last_active":"2024-03-29T04:09:55.539918+0000","last_peered":"2024-03-29T04:09:55.539918+0000","last_clean":"2024-03-29T04:09:55.539918+0000","last_became_active":"2024-03-29T04:09:53.022125+0000","last_became_peered":"2024-03-29T04:09:53.022125+0000","last_unstale":"2024-03-29T04:09:55.539918+0000","last_undegraded":"2024-03-29T04:09:55.539918+0000","last_fullsized":"2024-03-29T04:09:55.539918+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:34.192140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540347+0000","last_change":"2024-03-29T04:09:54.029372+0000","last_active":"2024-03-29T04:09:55.540347+0000","last_peered":"2024-03-29T04:09:55.540347+0000","last_clean":"2024-03-29T04:09:55.540347+0000","last_became_active":"2024-03-29T04:09:54.029077+0000","last_became_peered":"2024-03-29T04:09:54.029077+0000","last_unstale":"2024-03-29T04:09:55.540347+0000","last_undegraded":"2024-03-29T04:09:55.540347+0000","last_fullsized":"2024-03-29T04:09:55.540347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:37:51.878688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040454+0000","last_change":"2024-03-29T04:09:53.025830+0000","last_active":"2024-03-29T04:09:55.040454+0000","last_peered":"2024-03-29T04:09:55.040454+0000","last_clean":"2024-03-29T04:09:55.040454+0000","last_became_active":"2024-03-29T04:09:53.025466+0000","last_became_peered":"2024-03-29T04:09:53.025466+0000","last_unstale":"2024-03-29T04:09:55.040454+0000","last_undegraded":"2024-03-29T04:09:55.040454+0000","last_fullsized":"2024-03-29T04:09:55.040454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:36.980672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540506+0000","last_change":"2024-03-29T04:09:54.036408+0000","last_active":"2024-03-29T04:09:55.540506+0000","last_peered":"2024-03-29T04:09:55.540506+0000","last_clean":"2024-03-29T04:09:55.540506+0000","last_became_active":"2024-03-29T04:09:54.036042+0000","last_became_peered":"2024-03-29T04:09:54.036042+0000","last_unstale":"2024-03-29T04:09:55.540506+0000","last_undegraded":"2024-03-29T04:09:55.540506+0000","last_fullsized":"2024-03-29T04:09:55.540506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:34.175737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039209+0000","last_change":"2024-03-29T04:09:53.021983+0000","last_active":"2024-03-29T04:09:55.039209+0000","last_peered":"2024-03-29T04:09:55.039209+0000","last_clean":"2024-03-29T04:09:55.039209+0000","last_became_active":"2024-03-29T04:09:53.021717+0000","last_became_peered":"2024-03-29T04:09:53.021717+0000","last_unstale":"2024-03-29T04:09:55.039209+0000","last_undegraded":"2024-03-29T04:09:55.039209+0000","last_fullsized":"2024-03-29T04:09:55.039209+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:46.362685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.037961+0000","last_change":"2024-03-29T04:09:54.030131+0000","last_active":"2024-03-29T04:09:55.037961+0000","last_peered":"2024-03-29T04:09:55.037961+0000","last_clean":"2024-03-29T04:09:55.037961+0000","last_became_active":"2024-03-29T04:09:54.029830+0000","last_became_peered":"2024-03-29T04:09:54.029830+0000","last_unstale":"2024-03-29T04:09:55.037961+0000","last_undegraded":"2024-03-29T04:09:55.037961+0000","last_fullsized":"2024-03-29T04:09:55.037961+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.103608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040240+0000","last_change":"2024-03-29T04:09:53.022634+0000","last_active":"2024-03-29T04:09:55.040240+0000","last_peered":"2024-03-29T04:09:55.040240+0000","last_clean":"2024-03-29T04:09:55.040240+0000","last_became_active":"2024-03-29T04:09:53.022386+0000","last_became_peered":"2024-03-29T04:09:53.022386+0000","last_unstale":"2024-03-29T04:09:55.040240+0000","last_undegraded":"2024-03-29T04:09:55.040240+0000","last_fullsized":"2024-03-29T04:09:55.040240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:06.969462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039381+0000","last_change":"2024-03-29T04:09:54.028703+0000","last_active":"2024-03-29T04:09:55.039381+0000","last_peered":"2024-03-29T04:09:55.039381+0000","last_clean":"2024-03-29T04:09:55.039381+0000","last_became_active":"2024-03-29T04:09:54.028424+0000","last_became_peered":"2024-03-29T04:09:54.028424+0000","last_unstale":"2024-03-29T04:09:55.039381+0000","last_undegraded":"2024-03-29T04:09:55.039381+0000","last_fullsized":"2024-03-29T04:09:55.039381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:37.984666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038240+0000","last_change":"2024-03-29T04:09:53.017984+0000","last_active":"2024-03-29T04:09:55.038240+0000","last_peered":"2024-03-29T04:09:55.038240+0000","last_clean":"2024-03-29T04:09:55.038240+0000","last_became_active":"2024-03-29T04:09:53.017714+0000","last_became_peered":"2024-03-29T04:09:53.017714+0000","last_unstale":"2024-03-29T04:09:55.038240+0000","last_undegraded":"2024-03-29T04:09:55.038240+0000","last_fullsized":"2024-03-29T04:09:55.038240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:36.980672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038151+0000","last_change":"2024-03-29T04:09:54.040047+0000","last_active":"2024-03-29T04:09:55.038151+0000","last_peered":"2024-03-29T04:09:55.038151+0000","last_clean":"2024-03-29T04:09:55.038151+0000","last_became_active":"2024-03-29T04:09:54.039786+0000","last_became_peered":"2024-03-29T04:09:54.039786+0000","last_unstale":"2024-03-29T04:09:55.038151+0000","last_undegraded":"2024-03-29T04:09:55.038151+0000","last_fullsized":"2024-03-29T04:09:55.038151+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:51.065761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039930+0000","last_change":"2024-03-29T04:09:53.026336+0000","last_active":"2024-03-29T04:09:55.039930+0000","last_peered":"2024-03-29T04:09:55.039930+0000","last_clean":"2024-03-29T04:09:55.039930+0000","last_became_active":"2024-03-29T04:09:53.025584+0000","last_became_peered":"2024-03-29T04:09:53.025584+0000","last_unstale":"2024-03-29T04:09:55.039930+0000","last_undegraded":"2024-03-29T04:09:55.039930+0000","last_fullsized":"2024-03-29T04:09:55.039930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:40:36.681703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018629+0000","last_change":"2024-03-29T04:09:54.028882+0000","last_active":"2024-03-29T04:09:55.018629+0000","last_peered":"2024-03-29T04:09:55.018629+0000","last_clean":"2024-03-29T04:09:55.018629+0000","last_became_active":"2024-03-29T04:09:54.028596+0000","last_became_peered":"2024-03-29T04:09:54.028596+0000","last_unstale":"2024-03-29T04:09:55.018629+0000","last_undegraded":"2024-03-29T04:09:55.018629+0000","last_fullsized":"2024-03-29T04:09:55.018629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:23.490248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023242+0000","last_change":"2024-03-29T04:09:53.015771+0000","last_active":"2024-03-29T04:09:55.023242+0000","last_peered":"2024-03-29T04:09:55.023242+0000","last_clean":"2024-03-29T04:09:55.023242+0000","last_became_active":"2024-03-29T04:09:53.015436+0000","last_became_peered":"2024-03-29T04:09:53.015436+0000","last_unstale":"2024-03-29T04:09:55.023242+0000","last_undegraded":"2024-03-29T04:09:55.023242+0000","last_fullsized":"2024-03-29T04:09:55.023242+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:14.239068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023320+0000","last_change":"2024-03-29T04:09:54.038824+0000","last_active":"2024-03-29T04:09:55.023320+0000","last_peered":"2024-03-29T04:09:55.023320+0000","last_clean":"2024-03-29T04:09:55.023320+0000","last_became_active":"2024-03-29T04:09:54.038499+0000","last_became_peered":"2024-03-29T04:09:54.038499+0000","last_unstale":"2024-03-29T04:09:55.023320+0000","last_undegraded":"2024-03-29T04:09:55.023320+0000","last_fullsized":"2024-03-29T04:09:55.023320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:15.401317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018636+0000","last_change":"2024-03-29T04:09:53.025251+0000","last_active":"2024-03-29T04:09:55.018636+0000","last_peered":"2024-03-29T04:09:55.018636+0000","last_clean":"2024-03-29T04:09:55.018636+0000","last_became_active":"2024-03-29T04:09:53.024927+0000","last_became_peered":"2024-03-29T04:09:53.024927+0000","last_unstale":"2024-03-29T04:09:55.018636+0000","last_undegraded":"2024-03-29T04:09:55.018636+0000","last_fullsized":"2024-03-29T04:09:55.018636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:41.222810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.021953+0000","last_change":"2024-03-29T04:09:54.037954+0000","last_active":"2024-03-29T04:09:55.021953+0000","last_peered":"2024-03-29T04:09:55.021953+0000","last_clean":"2024-03-29T04:09:55.021953+0000","last_became_active":"2024-03-29T04:09:54.037709+0000","last_became_peered":"2024-03-29T04:09:54.037709+0000","last_unstale":"2024-03-29T04:09:55.021953+0000","last_undegraded":"2024-03-29T04:09:55.021953+0000","last_fullsized":"2024-03-29T04:09:55.021953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:38.793033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039328+0000","last_change":"2024-03-29T04:09:53.026838+0000","last_active":"2024-03-29T04:09:55.039328+0000","last_peered":"2024-03-29T04:09:55.039328+0000","last_clean":"2024-03-29T04:09:55.039328+0000","last_became_active":"2024-03-29T04:09:53.026509+0000","last_became_peered":"2024-03-29T04:09:53.026509+0000","last_unstale":"2024-03-29T04:09:55.039328+0000","last_undegraded":"2024-03-29T04:09:55.039328+0000","last_fullsized":"2024-03-29T04:09:55.039328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:27.437037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022677+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:09:55.022677+0000","last_peered":"2024-03-29T04:09:55.022677+0000","last_clean":"2024-03-29T04:09:55.022677+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:09:55.022677+0000","last_undegraded":"2024-03-29T04:09:55.022677+0000","last_fullsized":"2024-03-29T04:09:55.022677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542162,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":740617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542162,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":151793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":143929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":151793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":740527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:09:58.018 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:09:58.172 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.193+0000 7f46c4af2640 1 Processor -- start 2024-03-29T04:09:58.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.193+0000 7f46c4af2640 1 -- start start 2024-03-29T04:09:58.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46c4af2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f46c00af6b0 0x7f46c015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46c4af2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c00b3050 0x7f46c006f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46c4af2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c015c7e0 con 0x7f46c006f5d0 2024-03-29T04:09:58.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46c4af2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f46c0072760 con 0x7f46c00af6b0 2024-03-29T04:09:58.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46c4af2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f46c00afa90 con 0x7f46c00b3050 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f46c00af6b0 0x7f46c015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f46c00af6b0 0x7f46c015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35550/0 (socket says 172.21.15.156:35550) 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46bed9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f46c006f5d0 0x7f46c015ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44686/0 (socket says 172.21.15.156:44686) 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 -- 172.21.15.156:0/757452661 learned_addr learned my addr 172.21.15.156:0/757452661 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46bdd9b640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c00b3050 0x7f46c006f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 -- 172.21.15.156:0/757452661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c00b3050 msgr2=0x7f46c006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c00b3050 0x7f46c006f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 -- 172.21.15.156:0/757452661 >> v1:172.21.15.156:6789/0 conn(0x7f46c006f5d0 legacy=0x7f46c015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46bd59a640 1 -- 172.21.15.156:0/757452661 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163972121 0 0) 0x7f46b0001110 con 0x7f46c006f5d0 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46be59c640 1 -- 172.21.15.156:0/757452661 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c00b3510 con 0x7f46c00af6b0 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.194+0000 7f46bdd9b640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c00b3050 0x7f46c006f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.195+0000 7f46be59c640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015c2a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009870 tx=0x7f46b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41fc282e1b747194 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.195+0000 7f46bd59a640 1 -- 172.21.15.156:0/757452661 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8035020 con 0x7f46c00af6b0 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.195+0000 7f46bd59a640 1 -- 172.21.15.156:0/757452661 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b8031640 con 0x7f46c00af6b0 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.195+0000 7f46bd59a640 1 -- 172.21.15.156:0/757452661 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b802d040 con 0x7f46c00af6b0 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 -- 172.21.15.156:0/757452661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 msgr2=0x7f46c015c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015c2a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009870 tx=0x7f46b802b8f0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 -- 172.21.15.156:0/757452661 shutdown_connections 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c00b3050 0x7f46c006f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 --2- 172.21.15.156:0/757452661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015c2a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 -- 172.21.15.156:0/757452661 >> 172.21.15.156:0/757452661 conn(0x7f46c00b1d50 msgr2=0x7f46c00b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 -- 172.21.15.156:0/757452661 shutdown_connections 2024-03-29T04:09:58.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.196+0000 7f46c4af2640 1 -- 172.21.15.156:0/757452661 wait complete. 2024-03-29T04:09:58.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.197+0000 7f46c4af2640 1 Processor -- start 2024-03-29T04:09:58.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.197+0000 7f46c4af2640 1 -- start start 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46c4af2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c006f5d0 0x7f46c00ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46c4af2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46c4af2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c00b3050 0x7f46c015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46c4af2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46c00b3db0 con 0x7f46c00af6b0 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46c4af2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46c00b3ad0 con 0x7f46c00b3050 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46c4af2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f46c00b4090 con 0x7f46c006f5d0 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35564/0 (socket says 172.21.15.156:35564) 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 -- 172.21.15.156:0/2591550963 learned_addr learned my addr 172.21.15.156:0/2591550963 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46be59c640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c006f5d0 0x7f46c00ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bed9d640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c00b3050 0x7f46c015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 -- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c006f5d0 msgr2=0x7f46c00ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c006f5d0 0x7f46c00ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 -- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c00b3050 msgr2=0x7f46c015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c00b3050 0x7f46c015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bdd9b640 1 -- 172.21.15.156:0/2591550963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c01f5b70 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.198+0000 7f46bed9d640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c00b3050 0x7f46c015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.199+0000 7f46bdd9b640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015a1f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f46b400ce00 tx=0x7f46b400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.199+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b400f890 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.199+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b400fa30 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.199+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c01f5e50 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.199+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b400fe40 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.199+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c01f6410 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.200+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c0068610 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.200+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f46b4018400 con 0x7f46c00af6b0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.201+0000 7f46aeffd640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f46a006c560 0x7f46a006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.201+0000 7f46be59c640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f46a006c560 0x7f46a006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.201+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f46b408c580 con 0x7f46c00af6b0 2024-03-29T04:09:58.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.201+0000 7f46be59c640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f46a006c560 0x7f46a006ea20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f46b8036000 tx=0x7f46b8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.205+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f46b4014030 con 0x7f46c00af6b0 2024-03-29T04:09:58.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.385+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46c006da70 con 0x7f46a006c560 2024-03-29T04:09:58.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.391+0000 7f46aeffd640 1 -- 172.21.15.156:0/2591550963 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f46c006da70 con 0x7f46a006c560 2024-03-29T04:09:58.373 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:58.374 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f46a006c560 msgr2=0x7f46a006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f46a006c560 0x7f46a006ea20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f46b8036000 tx=0x7f46b8008030 comp rx=0 tx=0).stop 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 msgr2=0x7f46c015a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015a1f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f46b400ce00 tx=0x7f46b400a650 comp rx=0 tx=0).stop 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 shutdown_connections 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f46a006c560 0x7f46a006ea20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c00b3050 0x7f46c015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c00af6b0 0x7f46c015a1f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 --2- 172.21.15.156:0/2591550963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46c006f5d0 0x7f46c00ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 >> 172.21.15.156:0/2591550963 conn(0x7f46c00b1d50 msgr2=0x7f46c006df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 shutdown_connections 2024-03-29T04:09:58.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.395+0000 7f46c4af2640 1 -- 172.21.15.156:0/2591550963 wait complete. 2024-03-29T04:09:58.385 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-29T04:09:57.648583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237004,"kb_used_data":4188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737716,"statfs":{"total":773094113280,"available":772851421184,"internally_reserved":0,"allocated":4288512,"data_stored":2380528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.646651"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039499+0000","last_change":"2024-03-29T04:09:54.037102+0000","last_active":"2024-03-29T04:09:55.039499+0000","last_peered":"2024-03-29T04:09:55.039499+0000","last_clean":"2024-03-29T04:09:55.039499+0000","last_became_active":"2024-03-29T04:09:54.036788+0000","last_became_peered":"2024-03-29T04:09:54.036788+0000","last_unstale":"2024-03-29T04:09:55.039499+0000","last_undegraded":"2024-03-29T04:09:55.039499+0000","last_fullsized":"2024-03-29T04:09:55.039499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:24.718335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019400+0000","last_change":"2024-03-29T04:09:54.034758+0000","last_active":"2024-03-29T04:09:55.019400+0000","last_peered":"2024-03-29T04:09:55.019400+0000","last_clean":"2024-03-29T04:09:55.019400+0000","last_became_active":"2024-03-29T04:09:54.034482+0000","last_became_peered":"2024-03-29T04:09:54.034482+0000","last_unstale":"2024-03-29T04:09:55.019400+0000","last_undegraded":"2024-03-29T04:09:55.019400+0000","last_fullsized":"2024-03-29T04:09:55.019400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:48:24.105174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540258+0000","last_change":"2024-03-29T04:09:54.036045+0000","last_active":"2024-03-29T04:09:55.540258+0000","last_peered":"2024-03-29T04:09:55.540258+0000","last_clean":"2024-03-29T04:09:55.540258+0000","last_became_active":"2024-03-29T04:09:54.033549+0000","last_became_peered":"2024-03-29T04:09:54.033549+0000","last_unstale":"2024-03-29T04:09:55.540258+0000","last_undegraded":"2024-03-29T04:09:55.540258+0000","last_fullsized":"2024-03-29T04:09:55.540258+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:52.105155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040156+0000","last_change":"2024-03-29T04:09:54.031752+0000","last_active":"2024-03-29T04:09:55.040156+0000","last_peered":"2024-03-29T04:09:55.040156+0000","last_clean":"2024-03-29T04:09:55.040156+0000","last_became_active":"2024-03-29T04:09:54.031436+0000","last_became_peered":"2024-03-29T04:09:54.031436+0000","last_unstale":"2024-03-29T04:09:55.040156+0000","last_undegraded":"2024-03-29T04:09:55.040156+0000","last_fullsized":"2024-03-29T04:09:55.040156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:08.634564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540100+0000","last_change":"2024-03-29T04:09:54.036134+0000","last_active":"2024-03-29T04:09:55.540100+0000","last_peered":"2024-03-29T04:09:55.540100+0000","last_clean":"2024-03-29T04:09:55.540100+0000","last_became_active":"2024-03-29T04:09:54.035834+0000","last_became_peered":"2024-03-29T04:09:54.035834+0000","last_unstale":"2024-03-29T04:09:55.540100+0000","last_undegraded":"2024-03-29T04:09:55.540100+0000","last_fullsized":"2024-03-29T04:09:55.540100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.404392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018813+0000","last_change":"2024-03-29T04:09:54.038418+0000","last_active":"2024-03-29T04:09:55.018813+0000","last_peered":"2024-03-29T04:09:55.018813+0000","last_clean":"2024-03-29T04:09:55.018813+0000","last_became_active":"2024-03-29T04:09:54.038170+0000","last_became_peered":"2024-03-29T04:09:54.038170+0000","last_unstale":"2024-03-29T04:09:55.018813+0000","last_undegraded":"2024-03-29T04:09:55.018813+0000","last_fullsized":"2024-03-29T04:09:55.018813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:52.196258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022075+0000","last_change":"2024-03-29T04:09:54.042972+0000","last_active":"2024-03-29T04:09:55.022075+0000","last_peered":"2024-03-29T04:09:55.022075+0000","last_clean":"2024-03-29T04:09:55.022075+0000","last_became_active":"2024-03-29T04:09:54.042701+0000","last_became_peered":"2024-03-29T04:09:54.042701+0000","last_unstale":"2024-03-29T04:09:55.022075+0000","last_undegraded":"2024-03-29T04:09:55.022075+0000","last_fullsized":"2024-03-29T04:09:55.022075+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:55:57.600054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022123+0000","last_change":"2024-03-29T04:09:54.039996+0000","last_active":"2024-03-29T04:09:55.022123+0000","last_peered":"2024-03-29T04:09:55.022123+0000","last_clean":"2024-03-29T04:09:55.022123+0000","last_became_active":"2024-03-29T04:09:54.039712+0000","last_became_peered":"2024-03-29T04:09:54.039712+0000","last_unstale":"2024-03-29T04:09:55.022123+0000","last_undegraded":"2024-03-29T04:09:55.022123+0000","last_fullsized":"2024-03-29T04:09:55.022123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:27.334351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023068+0000","last_change":"2024-03-29T04:09:54.033289+0000","last_active":"2024-03-29T04:09:55.023068+0000","last_peered":"2024-03-29T04:09:55.023068+0000","last_clean":"2024-03-29T04:09:55.023068+0000","last_became_active":"2024-03-29T04:09:54.032168+0000","last_became_peered":"2024-03-29T04:09:54.032168+0000","last_unstale":"2024-03-29T04:09:55.023068+0000","last_undegraded":"2024-03-29T04:09:55.023068+0000","last_fullsized":"2024-03-29T04:09:55.023068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.103608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040287+0000","last_change":"2024-03-29T04:09:54.034286+0000","last_active":"2024-03-29T04:09:55.040287+0000","last_peered":"2024-03-29T04:09:55.040287+0000","last_clean":"2024-03-29T04:09:55.040287+0000","last_became_active":"2024-03-29T04:09:54.034031+0000","last_became_peered":"2024-03-29T04:09:54.034031+0000","last_unstale":"2024-03-29T04:09:55.040287+0000","last_undegraded":"2024-03-29T04:09:55.040287+0000","last_fullsized":"2024-03-29T04:09:55.040287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:58.350300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022004+0000","last_change":"2024-03-29T04:09:54.038944+0000","last_active":"2024-03-29T04:09:55.022004+0000","last_peered":"2024-03-29T04:09:55.022004+0000","last_clean":"2024-03-29T04:09:55.022004+0000","last_became_active":"2024-03-29T04:09:54.038672+0000","last_became_peered":"2024-03-29T04:09:54.038672+0000","last_unstale":"2024-03-29T04:09:55.022004+0000","last_undegraded":"2024-03-29T04:09:55.022004+0000","last_fullsized":"2024-03-29T04:09:55.022004+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:23.490248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540233+0000","last_change":"2024-03-29T04:09:54.034476+0000","last_active":"2024-03-29T04:09:55.540233+0000","last_peered":"2024-03-29T04:09:55.540233+0000","last_clean":"2024-03-29T04:09:55.540233+0000","last_became_active":"2024-03-29T04:09:54.034202+0000","last_became_peered":"2024-03-29T04:09:54.034202+0000","last_unstale":"2024-03-29T04:09:55.540233+0000","last_undegraded":"2024-03-29T04:09:55.540233+0000","last_fullsized":"2024-03-29T04:09:55.540233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:27.334351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540124+0000","last_change":"2024-03-29T04:09:54.025791+0000","last_active":"2024-03-29T04:09:55.540124+0000","last_peered":"2024-03-29T04:09:55.540124+0000","last_clean":"2024-03-29T04:09:55.540124+0000","last_became_active":"2024-03-29T04:09:54.025521+0000","last_became_peered":"2024-03-29T04:09:54.025521+0000","last_unstale":"2024-03-29T04:09:55.540124+0000","last_undegraded":"2024-03-29T04:09:55.540124+0000","last_fullsized":"2024-03-29T04:09:55.540124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:22.735465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018942+0000","last_change":"2024-03-29T04:09:54.027287+0000","last_active":"2024-03-29T04:09:55.018942+0000","last_peered":"2024-03-29T04:09:55.018942+0000","last_clean":"2024-03-29T04:09:55.018942+0000","last_became_active":"2024-03-29T04:09:54.026998+0000","last_became_peered":"2024-03-29T04:09:54.026998+0000","last_unstale":"2024-03-29T04:09:55.018942+0000","last_undegraded":"2024-03-29T04:09:55.018942+0000","last_fullsized":"2024-03-29T04:09:55.018942+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:58.930112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540066+0000","last_change":"2024-03-29T04:09:54.032319+0000","last_active":"2024-03-29T04:09:55.540066+0000","last_peered":"2024-03-29T04:09:55.540066+0000","last_clean":"2024-03-29T04:09:55.540066+0000","last_became_active":"2024-03-29T04:09:54.031957+0000","last_became_peered":"2024-03-29T04:09:54.031957+0000","last_unstale":"2024-03-29T04:09:55.540066+0000","last_undegraded":"2024-03-29T04:09:55.540066+0000","last_fullsized":"2024-03-29T04:09:55.540066+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:52.196258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022617+0000","last_change":"2024-03-29T04:09:54.044192+0000","last_active":"2024-03-29T04:09:55.022617+0000","last_peered":"2024-03-29T04:09:55.022617+0000","last_clean":"2024-03-29T04:09:55.022617+0000","last_became_active":"2024-03-29T04:09:54.043907+0000","last_became_peered":"2024-03-29T04:09:54.043907+0000","last_unstale":"2024-03-29T04:09:55.022617+0000","last_undegraded":"2024-03-29T04:09:55.022617+0000","last_fullsized":"2024-03-29T04:09:55.022617+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:15:50.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022436+0000","last_change":"2024-03-29T04:09:54.043341+0000","last_active":"2024-03-29T04:09:55.022436+0000","last_peered":"2024-03-29T04:09:55.022436+0000","last_clean":"2024-03-29T04:09:55.022436+0000","last_became_active":"2024-03-29T04:09:54.042757+0000","last_became_peered":"2024-03-29T04:09:54.042757+0000","last_unstale":"2024-03-29T04:09:55.022436+0000","last_undegraded":"2024-03-29T04:09:55.022436+0000","last_fullsized":"2024-03-29T04:09:55.022436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:48:24.105174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019515+0000","last_change":"2024-03-29T04:09:54.035421+0000","last_active":"2024-03-29T04:09:55.019515+0000","last_peered":"2024-03-29T04:09:55.019515+0000","last_clean":"2024-03-29T04:09:55.019515+0000","last_became_active":"2024-03-29T04:09:54.035160+0000","last_became_peered":"2024-03-29T04:09:54.035160+0000","last_unstale":"2024-03-29T04:09:55.019515+0000","last_undegraded":"2024-03-29T04:09:55.019515+0000","last_fullsized":"2024-03-29T04:09:55.019515+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:15:50.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018754+0000","last_change":"2024-03-29T04:09:54.034314+0000","last_active":"2024-03-29T04:09:55.018754+0000","last_peered":"2024-03-29T04:09:55.018754+0000","last_clean":"2024-03-29T04:09:55.018754+0000","last_became_active":"2024-03-29T04:09:54.034059+0000","last_became_peered":"2024-03-29T04:09:54.034059+0000","last_unstale":"2024-03-29T04:09:55.018754+0000","last_undegraded":"2024-03-29T04:09:55.018754+0000","last_fullsized":"2024-03-29T04:09:55.018754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:55:57.600054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540282+0000","last_change":"2024-03-29T04:09:54.030152+0000","last_active":"2024-03-29T04:09:55.540282+0000","last_peered":"2024-03-29T04:09:55.540282+0000","last_clean":"2024-03-29T04:09:55.540282+0000","last_became_active":"2024-03-29T04:09:54.029878+0000","last_became_peered":"2024-03-29T04:09:54.029878+0000","last_unstale":"2024-03-29T04:09:55.540282+0000","last_undegraded":"2024-03-29T04:09:55.540282+0000","last_fullsized":"2024-03-29T04:09:55.540282+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:15.243062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022972+0000","last_change":"2024-03-29T04:09:54.026691+0000","last_active":"2024-03-29T04:09:55.022972+0000","last_peered":"2024-03-29T04:09:55.022972+0000","last_clean":"2024-03-29T04:09:55.022972+0000","last_became_active":"2024-03-29T04:09:54.026254+0000","last_became_peered":"2024-03-29T04:09:54.026254+0000","last_unstale":"2024-03-29T04:09:55.022972+0000","last_undegraded":"2024-03-29T04:09:55.022972+0000","last_fullsized":"2024-03-29T04:09:55.022972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:42.226804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023249+0000","last_change":"2024-03-29T04:09:54.028422+0000","last_active":"2024-03-29T04:09:55.023249+0000","last_peered":"2024-03-29T04:09:55.023249+0000","last_clean":"2024-03-29T04:09:55.023249+0000","last_became_active":"2024-03-29T04:09:54.028128+0000","last_became_peered":"2024-03-29T04:09:54.028128+0000","last_unstale":"2024-03-29T04:09:55.023249+0000","last_undegraded":"2024-03-29T04:09:55.023249+0000","last_fullsized":"2024-03-29T04:09:55.023249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:16:37.027808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038118+0000","last_change":"2024-03-29T04:09:54.042812+0000","last_active":"2024-03-29T04:09:55.038118+0000","last_peered":"2024-03-29T04:09:55.038118+0000","last_clean":"2024-03-29T04:09:55.038118+0000","last_became_active":"2024-03-29T04:09:54.036389+0000","last_became_peered":"2024-03-29T04:09:54.036389+0000","last_unstale":"2024-03-29T04:09:55.038118+0000","last_undegraded":"2024-03-29T04:09:55.038118+0000","last_fullsized":"2024-03-29T04:09:55.038118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:35.196134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.539987+0000","last_change":"2024-03-29T04:09:54.028381+0000","last_active":"2024-03-29T04:09:55.539987+0000","last_peered":"2024-03-29T04:09:55.539987+0000","last_clean":"2024-03-29T04:09:55.539987+0000","last_became_active":"2024-03-29T04:09:54.028071+0000","last_became_peered":"2024-03-29T04:09:54.028071+0000","last_unstale":"2024-03-29T04:09:55.539987+0000","last_undegraded":"2024-03-29T04:09:55.539987+0000","last_fullsized":"2024-03-29T04:09:55.539987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:55:57.600054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040231+0000","last_change":"2024-03-29T04:09:54.038381+0000","last_active":"2024-03-29T04:09:55.040231+0000","last_peered":"2024-03-29T04:09:55.040231+0000","last_clean":"2024-03-29T04:09:55.040231+0000","last_became_active":"2024-03-29T04:09:54.038100+0000","last_became_peered":"2024-03-29T04:09:54.038100+0000","last_unstale":"2024-03-29T04:09:55.040231+0000","last_undegraded":"2024-03-29T04:09:55.040231+0000","last_fullsized":"2024-03-29T04:09:55.040231+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:13.319250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019602+0000","last_change":"2024-03-29T04:09:54.039126+0000","last_active":"2024-03-29T04:09:55.019602+0000","last_peered":"2024-03-29T04:09:55.019602+0000","last_clean":"2024-03-29T04:09:55.019602+0000","last_became_active":"2024-03-29T04:09:54.038872+0000","last_became_peered":"2024-03-29T04:09:54.038872+0000","last_unstale":"2024-03-29T04:09:55.019602+0000","last_undegraded":"2024-03-29T04:09:55.019602+0000","last_fullsized":"2024-03-29T04:09:55.019602+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:53:18.902650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040258+0000","last_change":"2024-03-29T04:09:54.031407+0000","last_active":"2024-03-29T04:09:55.040258+0000","last_peered":"2024-03-29T04:09:55.040258+0000","last_clean":"2024-03-29T04:09:55.040258+0000","last_became_active":"2024-03-29T04:09:54.031115+0000","last_became_peered":"2024-03-29T04:09:54.031115+0000","last_unstale":"2024-03-29T04:09:55.040258+0000","last_undegraded":"2024-03-29T04:09:55.040258+0000","last_fullsized":"2024-03-29T04:09:55.040258+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:32.934535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540348+0000","last_change":"2024-03-29T04:09:54.027543+0000","last_active":"2024-03-29T04:09:55.540348+0000","last_peered":"2024-03-29T04:09:55.540348+0000","last_clean":"2024-03-29T04:09:55.540348+0000","last_became_active":"2024-03-29T04:09:54.027208+0000","last_became_peered":"2024-03-29T04:09:54.027208+0000","last_unstale":"2024-03-29T04:09:55.540348+0000","last_undegraded":"2024-03-29T04:09:55.540348+0000","last_fullsized":"2024-03-29T04:09:55.540348+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:48:24.105174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023342+0000","last_change":"2024-03-29T04:09:54.038213+0000","last_active":"2024-03-29T04:09:55.023342+0000","last_peered":"2024-03-29T04:09:55.023342+0000","last_clean":"2024-03-29T04:09:55.023342+0000","last_became_active":"2024-03-29T04:09:54.037941+0000","last_became_peered":"2024-03-29T04:09:54.037941+0000","last_unstale":"2024-03-29T04:09:55.023342+0000","last_undegraded":"2024-03-29T04:09:55.023342+0000","last_fullsized":"2024-03-29T04:09:55.023342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:45.126080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.021846+0000","last_change":"2024-03-29T04:09:54.042301+0000","last_active":"2024-03-29T04:09:55.021846+0000","last_peered":"2024-03-29T04:09:55.021846+0000","last_clean":"2024-03-29T04:09:55.021846+0000","last_became_active":"2024-03-29T04:09:54.041984+0000","last_became_peered":"2024-03-29T04:09:54.041984+0000","last_unstale":"2024-03-29T04:09:55.021846+0000","last_undegraded":"2024-03-29T04:09:55.021846+0000","last_fullsized":"2024-03-29T04:09:55.021846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:58.350300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039318+0000","last_change":"2024-03-29T04:09:54.031350+0000","last_active":"2024-03-29T04:09:55.039318+0000","last_peered":"2024-03-29T04:09:55.039318+0000","last_clean":"2024-03-29T04:09:55.039318+0000","last_became_active":"2024-03-29T04:09:54.030955+0000","last_became_peered":"2024-03-29T04:09:54.030955+0000","last_unstale":"2024-03-29T04:09:55.039318+0000","last_undegraded":"2024-03-29T04:09:55.039318+0000","last_fullsized":"2024-03-29T04:09:55.039318+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:47.366680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022466+0000","last_change":"2024-03-29T04:09:54.036526+0000","last_active":"2024-03-29T04:09:55.022466+0000","last_peered":"2024-03-29T04:09:55.022466+0000","last_clean":"2024-03-29T04:09:55.022466+0000","last_became_active":"2024-03-29T04:09:54.036185+0000","last_became_peered":"2024-03-29T04:09:54.036185+0000","last_unstale":"2024-03-29T04:09:55.022466+0000","last_undegraded":"2024-03-29T04:09:55.022466+0000","last_fullsized":"2024-03-29T04:09:55.022466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:16:21.172007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038018+0000","last_change":"2024-03-29T04:09:54.035509+0000","last_active":"2024-03-29T04:09:55.038018+0000","last_peered":"2024-03-29T04:09:55.038018+0000","last_clean":"2024-03-29T04:09:55.038018+0000","last_became_active":"2024-03-29T04:09:54.035214+0000","last_became_peered":"2024-03-29T04:09:54.035214+0000","last_unstale":"2024-03-29T04:09:55.038018+0000","last_undegraded":"2024-03-29T04:09:55.038018+0000","last_fullsized":"2024-03-29T04:09:55.038018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:42.226804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039560+0000","last_change":"2024-03-29T04:09:53.028108+0000","last_active":"2024-03-29T04:09:55.039560+0000","last_peered":"2024-03-29T04:09:55.039560+0000","last_clean":"2024-03-29T04:09:55.039560+0000","last_became_active":"2024-03-29T04:09:53.027794+0000","last_became_peered":"2024-03-29T04:09:53.027794+0000","last_unstale":"2024-03-29T04:09:55.039560+0000","last_undegraded":"2024-03-29T04:09:55.039560+0000","last_fullsized":"2024-03-29T04:09:55.039560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:51.906050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039531+0000","last_change":"2024-03-29T04:09:54.035684+0000","last_active":"2024-03-29T04:09:55.039531+0000","last_peered":"2024-03-29T04:09:55.039531+0000","last_clean":"2024-03-29T04:09:55.039531+0000","last_became_active":"2024-03-29T04:09:54.035407+0000","last_became_peered":"2024-03-29T04:09:54.035407+0000","last_unstale":"2024-03-29T04:09:55.039531+0000","last_undegraded":"2024-03-29T04:09:55.039531+0000","last_fullsized":"2024-03-29T04:09:55.039531+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:20.156066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.037941+0000","last_change":"2024-03-29T04:09:53.019049+0000","last_active":"2024-03-29T04:09:55.037941+0000","last_peered":"2024-03-29T04:09:55.037941+0000","last_clean":"2024-03-29T04:09:55.037941+0000","last_became_active":"2024-03-29T04:09:53.018782+0000","last_became_peered":"2024-03-29T04:09:53.018782+0000","last_unstale":"2024-03-29T04:09:55.037941+0000","last_undegraded":"2024-03-29T04:09:55.037941+0000","last_fullsized":"2024-03-29T04:09:55.037941+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:02.067736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023142+0000","last_change":"2024-03-29T04:09:54.041127+0000","last_active":"2024-03-29T04:09:55.023142+0000","last_peered":"2024-03-29T04:09:55.023142+0000","last_clean":"2024-03-29T04:09:55.023142+0000","last_became_active":"2024-03-29T04:09:54.040801+0000","last_became_peered":"2024-03-29T04:09:54.040801+0000","last_unstale":"2024-03-29T04:09:55.023142+0000","last_undegraded":"2024-03-29T04:09:55.023142+0000","last_fullsized":"2024-03-29T04:09:55.023142+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:51.065761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018846+0000","last_change":"2024-03-29T04:09:53.017697+0000","last_active":"2024-03-29T04:09:55.018846+0000","last_peered":"2024-03-29T04:09:55.018846+0000","last_clean":"2024-03-29T04:09:55.018846+0000","last_became_active":"2024-03-29T04:09:53.017395+0000","last_became_peered":"2024-03-29T04:09:53.017395+0000","last_unstale":"2024-03-29T04:09:55.018846+0000","last_undegraded":"2024-03-29T04:09:55.018846+0000","last_fullsized":"2024-03-29T04:09:55.018846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:23.714341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040296+0000","last_change":"2024-03-29T04:09:54.029204+0000","last_active":"2024-03-29T04:09:55.040296+0000","last_peered":"2024-03-29T04:09:55.040296+0000","last_clean":"2024-03-29T04:09:55.040296+0000","last_became_active":"2024-03-29T04:09:54.028909+0000","last_became_peered":"2024-03-29T04:09:54.028909+0000","last_unstale":"2024-03-29T04:09:55.040296+0000","last_undegraded":"2024-03-29T04:09:55.040296+0000","last_fullsized":"2024-03-29T04:09:55.040296+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:38.793033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540233+0000","last_change":"2024-03-29T04:09:53.023779+0000","last_active":"2024-03-29T04:09:55.540233+0000","last_peered":"2024-03-29T04:09:55.540233+0000","last_clean":"2024-03-29T04:09:55.540233+0000","last_became_active":"2024-03-29T04:09:53.023516+0000","last_became_peered":"2024-03-29T04:09:53.023516+0000","last_unstale":"2024-03-29T04:09:55.540233+0000","last_undegraded":"2024-03-29T04:09:55.540233+0000","last_fullsized":"2024-03-29T04:09:55.540233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:01.927119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022036+0000","last_change":"2024-03-29T04:09:54.036519+0000","last_active":"2024-03-29T04:09:55.022036+0000","last_peered":"2024-03-29T04:09:55.022036+0000","last_clean":"2024-03-29T04:09:55.022036+0000","last_became_active":"2024-03-29T04:09:54.036200+0000","last_became_peered":"2024-03-29T04:09:54.036200+0000","last_unstale":"2024-03-29T04:09:55.022036+0000","last_undegraded":"2024-03-29T04:09:55.022036+0000","last_fullsized":"2024-03-29T04:09:55.022036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:52.196258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023443+0000","last_change":"2024-03-29T04:09:53.023448+0000","last_active":"2024-03-29T04:09:55.023443+0000","last_peered":"2024-03-29T04:09:55.023443+0000","last_clean":"2024-03-29T04:09:55.023443+0000","last_became_active":"2024-03-29T04:09:53.023020+0000","last_became_peered":"2024-03-29T04:09:53.023020+0000","last_unstale":"2024-03-29T04:09:55.023443+0000","last_undegraded":"2024-03-29T04:09:55.023443+0000","last_fullsized":"2024-03-29T04:09:55.023443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:27.437037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540119+0000","last_change":"2024-03-29T04:09:54.027552+0000","last_active":"2024-03-29T04:09:55.540119+0000","last_peered":"2024-03-29T04:09:55.540119+0000","last_clean":"2024-03-29T04:09:55.540119+0000","last_became_active":"2024-03-29T04:09:54.027266+0000","last_became_peered":"2024-03-29T04:09:54.027266+0000","last_unstale":"2024-03-29T04:09:55.540119+0000","last_undegraded":"2024-03-29T04:09:55.540119+0000","last_fullsized":"2024-03-29T04:09:55.540119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:58.930112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022896+0000","last_change":"2024-03-29T04:09:53.019184+0000","last_active":"2024-03-29T04:09:55.022896+0000","last_peered":"2024-03-29T04:09:55.022896+0000","last_clean":"2024-03-29T04:09:55.022896+0000","last_became_active":"2024-03-29T04:09:53.018866+0000","last_became_peered":"2024-03-29T04:09:53.018866+0000","last_unstale":"2024-03-29T04:09:55.022896+0000","last_undegraded":"2024-03-29T04:09:55.022896+0000","last_fullsized":"2024-03-29T04:09:55.022896+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:17.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022533+0000","last_change":"2024-03-29T04:09:54.031799+0000","last_active":"2024-03-29T04:09:55.022533+0000","last_peered":"2024-03-29T04:09:55.022533+0000","last_clean":"2024-03-29T04:09:55.022533+0000","last_became_active":"2024-03-29T04:09:54.031530+0000","last_became_peered":"2024-03-29T04:09:54.031530+0000","last_unstale":"2024-03-29T04:09:55.022533+0000","last_undegraded":"2024-03-29T04:09:55.022533+0000","last_fullsized":"2024-03-29T04:09:55.022533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:20.156066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018657+0000","last_change":"2024-03-29T04:09:53.018030+0000","last_active":"2024-03-29T04:09:55.018657+0000","last_peered":"2024-03-29T04:09:55.018657+0000","last_clean":"2024-03-29T04:09:55.018657+0000","last_became_active":"2024-03-29T04:09:53.017329+0000","last_became_peered":"2024-03-29T04:09:53.017329+0000","last_unstale":"2024-03-29T04:09:55.018657+0000","last_undegraded":"2024-03-29T04:09:55.018657+0000","last_fullsized":"2024-03-29T04:09:55.018657+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:16:36.023814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039773+0000","last_change":"2024-03-29T04:09:54.031093+0000","last_active":"2024-03-29T04:09:55.039773+0000","last_peered":"2024-03-29T04:09:55.039773+0000","last_clean":"2024-03-29T04:09:55.039773+0000","last_became_active":"2024-03-29T04:09:54.030685+0000","last_became_peered":"2024-03-29T04:09:54.030685+0000","last_unstale":"2024-03-29T04:09:55.039773+0000","last_undegraded":"2024-03-29T04:09:55.039773+0000","last_fullsized":"2024-03-29T04:09:55.039773+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:45.126080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022658+0000","last_change":"2024-03-29T04:09:53.022641+0000","last_active":"2024-03-29T04:09:55.022658+0000","last_peered":"2024-03-29T04:09:55.022658+0000","last_clean":"2024-03-29T04:09:55.022658+0000","last_became_active":"2024-03-29T04:09:53.022338+0000","last_became_peered":"2024-03-29T04:09:53.022338+0000","last_unstale":"2024-03-29T04:09:55.022658+0000","last_undegraded":"2024-03-29T04:09:55.022658+0000","last_fullsized":"2024-03-29T04:09:55.022658+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:51.246947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540414+0000","last_change":"2024-03-29T04:09:54.030623+0000","last_active":"2024-03-29T04:09:55.540414+0000","last_peered":"2024-03-29T04:09:55.540414+0000","last_clean":"2024-03-29T04:09:55.540414+0000","last_became_active":"2024-03-29T04:09:54.030352+0000","last_became_peered":"2024-03-29T04:09:54.030352+0000","last_unstale":"2024-03-29T04:09:55.540414+0000","last_undegraded":"2024-03-29T04:09:55.540414+0000","last_fullsized":"2024-03-29T04:09:55.540414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:15:50.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019138+0000","last_change":"2024-03-29T04:09:53.024582+0000","last_active":"2024-03-29T04:09:55.019138+0000","last_peered":"2024-03-29T04:09:55.019138+0000","last_clean":"2024-03-29T04:09:55.019138+0000","last_became_active":"2024-03-29T04:09:53.024238+0000","last_became_peered":"2024-03-29T04:09:53.024238+0000","last_unstale":"2024-03-29T04:09:55.019138+0000","last_undegraded":"2024-03-29T04:09:55.019138+0000","last_fullsized":"2024-03-29T04:09:55.019138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:34.192140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019054+0000","last_change":"2024-03-29T04:09:54.036761+0000","last_active":"2024-03-29T04:09:55.019054+0000","last_peered":"2024-03-29T04:09:55.019054+0000","last_clean":"2024-03-29T04:09:55.019054+0000","last_became_active":"2024-03-29T04:09:54.036503+0000","last_became_peered":"2024-03-29T04:09:54.036503+0000","last_unstale":"2024-03-29T04:09:55.019054+0000","last_undegraded":"2024-03-29T04:09:55.019054+0000","last_fullsized":"2024-03-29T04:09:55.019054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:52.105155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038072+0000","last_change":"2024-03-29T04:09:53.016622+0000","last_active":"2024-03-29T04:09:55.038072+0000","last_peered":"2024-03-29T04:09:55.038072+0000","last_clean":"2024-03-29T04:09:55.038072+0000","last_became_active":"2024-03-29T04:09:53.016288+0000","last_became_peered":"2024-03-29T04:09:53.016288+0000","last_unstale":"2024-03-29T04:09:55.038072+0000","last_undegraded":"2024-03-29T04:09:55.038072+0000","last_fullsized":"2024-03-29T04:09:55.038072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:40:36.681703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018968+0000","last_change":"2024-03-29T04:09:54.034065+0000","last_active":"2024-03-29T04:09:55.018968+0000","last_peered":"2024-03-29T04:09:55.018968+0000","last_clean":"2024-03-29T04:09:55.018968+0000","last_became_active":"2024-03-29T04:09:54.033795+0000","last_became_peered":"2024-03-29T04:09:54.033795+0000","last_unstale":"2024-03-29T04:09:55.018968+0000","last_undegraded":"2024-03-29T04:09:55.018968+0000","last_fullsized":"2024-03-29T04:09:55.018968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:27.334351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540405+0000","last_change":"2024-03-29T04:09:53.019135+0000","last_active":"2024-03-29T04:09:55.540405+0000","last_peered":"2024-03-29T04:09:55.540405+0000","last_clean":"2024-03-29T04:09:55.540405+0000","last_became_active":"2024-03-29T04:09:53.018856+0000","last_became_peered":"2024-03-29T04:09:53.018856+0000","last_unstale":"2024-03-29T04:09:55.540405+0000","last_undegraded":"2024-03-29T04:09:55.540405+0000","last_fullsized":"2024-03-29T04:09:55.540405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:50.061767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540295+0000","last_change":"2024-03-29T04:09:54.034651+0000","last_active":"2024-03-29T04:09:55.540295+0000","last_peered":"2024-03-29T04:09:55.540295+0000","last_clean":"2024-03-29T04:09:55.540295+0000","last_became_active":"2024-03-29T04:09:54.033682+0000","last_became_peered":"2024-03-29T04:09:54.033682+0000","last_unstale":"2024-03-29T04:09:55.540295+0000","last_undegraded":"2024-03-29T04:09:55.540295+0000","last_fullsized":"2024-03-29T04:09:55.540295+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:16:21.172007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019050+0000","last_change":"2024-03-29T04:09:53.018264+0000","last_active":"2024-03-29T04:09:55.019050+0000","last_peered":"2024-03-29T04:09:55.019050+0000","last_clean":"2024-03-29T04:09:55.019050+0000","last_became_active":"2024-03-29T04:09:53.017056+0000","last_became_peered":"2024-03-29T04:09:53.017056+0000","last_unstale":"2024-03-29T04:09:55.019050+0000","last_undegraded":"2024-03-29T04:09:55.019050+0000","last_fullsized":"2024-03-29T04:09:55.019050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:19.152072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039798+0000","last_change":"2024-03-29T04:09:54.039934+0000","last_active":"2024-03-29T04:09:55.039798+0000","last_peered":"2024-03-29T04:09:55.039798+0000","last_clean":"2024-03-29T04:09:55.039798+0000","last_became_active":"2024-03-29T04:09:54.039669+0000","last_became_peered":"2024-03-29T04:09:54.039669+0000","last_unstale":"2024-03-29T04:09:55.039798+0000","last_undegraded":"2024-03-29T04:09:55.039798+0000","last_fullsized":"2024-03-29T04:09:55.039798+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:45:58.678448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023541+0000","last_change":"2024-03-29T04:09:53.027098+0000","last_active":"2024-03-29T04:09:55.023541+0000","last_peered":"2024-03-29T04:09:55.023541+0000","last_clean":"2024-03-29T04:09:55.023541+0000","last_became_active":"2024-03-29T04:09:53.026801+0000","last_became_peered":"2024-03-29T04:09:53.026801+0000","last_unstale":"2024-03-29T04:09:55.023541+0000","last_undegraded":"2024-03-29T04:09:55.023541+0000","last_fullsized":"2024-03-29T04:09:55.023541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:21.731470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.019298+0000","last_change":"2024-03-29T04:09:54.033442+0000","last_active":"2024-03-29T04:09:55.019298+0000","last_peered":"2024-03-29T04:09:55.019298+0000","last_clean":"2024-03-29T04:09:55.019298+0000","last_became_active":"2024-03-29T04:09:54.033138+0000","last_became_peered":"2024-03-29T04:09:54.033138+0000","last_unstale":"2024-03-29T04:09:55.019298+0000","last_undegraded":"2024-03-29T04:09:55.019298+0000","last_fullsized":"2024-03-29T04:09:55.019298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:16:21.172007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039898+0000","last_change":"2024-03-29T04:09:53.025274+0000","last_active":"2024-03-29T04:09:55.039898+0000","last_peered":"2024-03-29T04:09:55.039898+0000","last_clean":"2024-03-29T04:09:55.039898+0000","last_became_active":"2024-03-29T04:09:53.025002+0000","last_became_peered":"2024-03-29T04:09:53.025002+0000","last_unstale":"2024-03-29T04:09:55.039898+0000","last_undegraded":"2024-03-29T04:09:55.039898+0000","last_fullsized":"2024-03-29T04:09:55.039898+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:02.067736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022269+0000","last_change":"2024-03-29T04:09:54.037254+0000","last_active":"2024-03-29T04:09:55.022269+0000","last_peered":"2024-03-29T04:09:55.022269+0000","last_clean":"2024-03-29T04:09:55.022269+0000","last_became_active":"2024-03-29T04:09:54.036970+0000","last_became_peered":"2024-03-29T04:09:54.036970+0000","last_unstale":"2024-03-29T04:09:55.022269+0000","last_undegraded":"2024-03-29T04:09:55.022269+0000","last_fullsized":"2024-03-29T04:09:55.022269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:52.105155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018598+0000","last_change":"2024-03-29T04:09:53.025284+0000","last_active":"2024-03-29T04:09:55.018598+0000","last_peered":"2024-03-29T04:09:55.018598+0000","last_clean":"2024-03-29T04:09:55.018598+0000","last_became_active":"2024-03-29T04:09:53.025018+0000","last_became_peered":"2024-03-29T04:09:53.025018+0000","last_unstale":"2024-03-29T04:09:55.018598+0000","last_undegraded":"2024-03-29T04:09:55.018598+0000","last_fullsized":"2024-03-29T04:09:55.018598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:06.969462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039442+0000","last_change":"2024-03-29T04:09:54.036284+0000","last_active":"2024-03-29T04:09:55.039442+0000","last_peered":"2024-03-29T04:09:55.039442+0000","last_clean":"2024-03-29T04:09:55.039442+0000","last_became_active":"2024-03-29T04:09:54.035804+0000","last_became_peered":"2024-03-29T04:09:54.035804+0000","last_unstale":"2024-03-29T04:09:55.039442+0000","last_undegraded":"2024-03-29T04:09:55.039442+0000","last_fullsized":"2024-03-29T04:09:55.039442+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:07.973456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018732+0000","last_change":"2024-03-29T04:09:53.016095+0000","last_active":"2024-03-29T04:09:55.018732+0000","last_peered":"2024-03-29T04:09:55.018732+0000","last_clean":"2024-03-29T04:09:55.018732+0000","last_became_active":"2024-03-29T04:09:53.015800+0000","last_became_peered":"2024-03-29T04:09:53.015800+0000","last_unstale":"2024-03-29T04:09:55.018732+0000","last_undegraded":"2024-03-29T04:09:55.018732+0000","last_fullsized":"2024-03-29T04:09:55.018732+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:23.099613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023061+0000","last_change":"2024-03-29T04:09:54.028344+0000","last_active":"2024-03-29T04:09:55.023061+0000","last_peered":"2024-03-29T04:09:55.023061+0000","last_clean":"2024-03-29T04:09:55.023061+0000","last_became_active":"2024-03-29T04:09:54.028063+0000","last_became_peered":"2024-03-29T04:09:54.028063+0000","last_unstale":"2024-03-29T04:09:55.023061+0000","last_undegraded":"2024-03-29T04:09:55.023061+0000","last_fullsized":"2024-03-29T04:09:55.023061+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:35.196134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040418+0000","last_change":"2024-03-29T04:09:53.021619+0000","last_active":"2024-03-29T04:09:55.040418+0000","last_peered":"2024-03-29T04:09:55.040418+0000","last_clean":"2024-03-29T04:09:55.040418+0000","last_became_active":"2024-03-29T04:09:53.021319+0000","last_became_peered":"2024-03-29T04:09:53.021319+0000","last_unstale":"2024-03-29T04:09:55.040418+0000","last_undegraded":"2024-03-29T04:09:55.040418+0000","last_fullsized":"2024-03-29T04:09:55.040418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:46.362685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540468+0000","last_change":"2024-03-29T04:09:54.034324+0000","last_active":"2024-03-29T04:09:55.540468+0000","last_peered":"2024-03-29T04:09:55.540468+0000","last_clean":"2024-03-29T04:09:55.540468+0000","last_became_active":"2024-03-29T04:09:54.033722+0000","last_became_peered":"2024-03-29T04:09:54.033722+0000","last_unstale":"2024-03-29T04:09:55.540468+0000","last_undegraded":"2024-03-29T04:09:55.540468+0000","last_fullsized":"2024-03-29T04:09:55.540468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:53:18.902650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022953+0000","last_change":"2024-03-29T04:09:53.027071+0000","last_active":"2024-03-29T04:09:55.022953+0000","last_peered":"2024-03-29T04:09:55.022953+0000","last_clean":"2024-03-29T04:09:55.022953+0000","last_became_active":"2024-03-29T04:09:53.026745+0000","last_became_peered":"2024-03-29T04:09:53.026745+0000","last_unstale":"2024-03-29T04:09:55.022953+0000","last_undegraded":"2024-03-29T04:09:55.022953+0000","last_fullsized":"2024-03-29T04:09:55.022953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:05.400397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040119+0000","last_change":"2024-03-29T04:09:54.029894+0000","last_active":"2024-03-29T04:09:55.040119+0000","last_peered":"2024-03-29T04:09:55.040119+0000","last_clean":"2024-03-29T04:09:55.040119+0000","last_became_active":"2024-03-29T04:09:54.029626+0000","last_became_peered":"2024-03-29T04:09:54.029626+0000","last_unstale":"2024-03-29T04:09:55.040119+0000","last_undegraded":"2024-03-29T04:09:55.040119+0000","last_fullsized":"2024-03-29T04:09:55.040119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:33:12.111409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023237+0000","last_change":"2024-03-29T04:09:53.017139+0000","last_active":"2024-03-29T04:09:55.023237+0000","last_peered":"2024-03-29T04:09:55.023237+0000","last_clean":"2024-03-29T04:09:55.023237+0000","last_became_active":"2024-03-29T04:09:53.016832+0000","last_became_peered":"2024-03-29T04:09:53.016832+0000","last_unstale":"2024-03-29T04:09:55.023237+0000","last_undegraded":"2024-03-29T04:09:55.023237+0000","last_fullsized":"2024-03-29T04:09:55.023237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:41.712929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540302+0000","last_change":"2024-03-29T04:09:54.036616+0000","last_active":"2024-03-29T04:09:55.540302+0000","last_peered":"2024-03-29T04:09:55.540302+0000","last_clean":"2024-03-29T04:09:55.540302+0000","last_became_active":"2024-03-29T04:09:54.036325+0000","last_became_peered":"2024-03-29T04:09:54.036325+0000","last_unstale":"2024-03-29T04:09:55.540302+0000","last_undegraded":"2024-03-29T04:09:55.540302+0000","last_fullsized":"2024-03-29T04:09:55.540302+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:49.771990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023265+0000","last_change":"2024-03-29T04:09:53.019125+0000","last_active":"2024-03-29T04:09:55.023265+0000","last_peered":"2024-03-29T04:09:55.023265+0000","last_clean":"2024-03-29T04:09:55.023265+0000","last_became_active":"2024-03-29T04:09:53.018823+0000","last_became_peered":"2024-03-29T04:09:53.018823+0000","last_unstale":"2024-03-29T04:09:55.023265+0000","last_undegraded":"2024-03-29T04:09:55.023265+0000","last_fullsized":"2024-03-29T04:09:55.023265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:42.552906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540093+0000","last_change":"2024-03-29T04:09:54.031341+0000","last_active":"2024-03-29T04:09:55.540093+0000","last_peered":"2024-03-29T04:09:55.540093+0000","last_clean":"2024-03-29T04:09:55.540093+0000","last_became_active":"2024-03-29T04:09:54.031071+0000","last_became_peered":"2024-03-29T04:09:54.031071+0000","last_unstale":"2024-03-29T04:09:55.540093+0000","last_undegraded":"2024-03-29T04:09:55.540093+0000","last_fullsized":"2024-03-29T04:09:55.540093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:52.250941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038105+0000","last_change":"2024-03-29T04:09:53.023196+0000","last_active":"2024-03-29T04:09:55.038105+0000","last_peered":"2024-03-29T04:09:55.038105+0000","last_clean":"2024-03-29T04:09:55.038105+0000","last_became_active":"2024-03-29T04:09:53.022902+0000","last_became_peered":"2024-03-29T04:09:53.022902+0000","last_unstale":"2024-03-29T04:09:55.038105+0000","last_undegraded":"2024-03-29T04:09:55.038105+0000","last_fullsized":"2024-03-29T04:09:55.038105+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:49.727836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540243+0000","last_change":"2024-03-29T04:09:54.023943+0000","last_active":"2024-03-29T04:09:55.540243+0000","last_peered":"2024-03-29T04:09:55.540243+0000","last_clean":"2024-03-29T04:09:55.540243+0000","last_became_active":"2024-03-29T04:09:54.023692+0000","last_became_peered":"2024-03-29T04:09:54.023692+0000","last_unstale":"2024-03-29T04:09:55.540243+0000","last_undegraded":"2024-03-29T04:09:55.540243+0000","last_fullsized":"2024-03-29T04:09:55.540243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:52.910044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039387+0000","last_change":"2024-03-29T04:09:53.027847+0000","last_active":"2024-03-29T04:09:55.039387+0000","last_peered":"2024-03-29T04:09:55.039387+0000","last_clean":"2024-03-29T04:09:55.039387+0000","last_became_active":"2024-03-29T04:09:53.027496+0000","last_became_peered":"2024-03-29T04:09:53.027496+0000","last_unstale":"2024-03-29T04:09:55.039387+0000","last_undegraded":"2024-03-29T04:09:55.039387+0000","last_fullsized":"2024-03-29T04:09:55.039387+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:21.731470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540108+0000","last_change":"2024-03-29T04:09:54.023583+0000","last_active":"2024-03-29T04:09:55.540108+0000","last_peered":"2024-03-29T04:09:55.540108+0000","last_clean":"2024-03-29T04:09:55.540108+0000","last_became_active":"2024-03-29T04:09:54.023273+0000","last_became_peered":"2024-03-29T04:09:54.023273+0000","last_unstale":"2024-03-29T04:09:55.540108+0000","last_undegraded":"2024-03-29T04:09:55.540108+0000","last_fullsized":"2024-03-29T04:09:55.540108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:28.441032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023033+0000","last_change":"2024-03-29T04:09:53.020050+0000","last_active":"2024-03-29T04:09:55.023033+0000","last_peered":"2024-03-29T04:09:55.023033+0000","last_clean":"2024-03-29T04:09:55.023033+0000","last_became_active":"2024-03-29T04:09:53.019803+0000","last_became_peered":"2024-03-29T04:09:53.019803+0000","last_unstale":"2024-03-29T04:09:55.023033+0000","last_undegraded":"2024-03-29T04:09:55.023033+0000","last_fullsized":"2024-03-29T04:09:55.023033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:51.906050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022077+0000","last_change":"2024-03-29T04:09:54.042188+0000","last_active":"2024-03-29T04:09:55.022077+0000","last_peered":"2024-03-29T04:09:55.022077+0000","last_clean":"2024-03-29T04:09:55.022077+0000","last_became_active":"2024-03-29T04:09:54.041896+0000","last_became_peered":"2024-03-29T04:09:54.041896+0000","last_unstale":"2024-03-29T04:09:55.022077+0000","last_undegraded":"2024-03-29T04:09:55.022077+0000","last_fullsized":"2024-03-29T04:09:55.022077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:58.930112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.539918+0000","last_change":"2024-03-29T04:09:53.022415+0000","last_active":"2024-03-29T04:09:55.539918+0000","last_peered":"2024-03-29T04:09:55.539918+0000","last_clean":"2024-03-29T04:09:55.539918+0000","last_became_active":"2024-03-29T04:09:53.022125+0000","last_became_peered":"2024-03-29T04:09:53.022125+0000","last_unstale":"2024-03-29T04:09:55.539918+0000","last_undegraded":"2024-03-29T04:09:55.539918+0000","last_fullsized":"2024-03-29T04:09:55.539918+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:34.192140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540347+0000","last_change":"2024-03-29T04:09:54.029372+0000","last_active":"2024-03-29T04:09:55.540347+0000","last_peered":"2024-03-29T04:09:55.540347+0000","last_clean":"2024-03-29T04:09:55.540347+0000","last_became_active":"2024-03-29T04:09:54.029077+0000","last_became_peered":"2024-03-29T04:09:54.029077+0000","last_unstale":"2024-03-29T04:09:55.540347+0000","last_undegraded":"2024-03-29T04:09:55.540347+0000","last_fullsized":"2024-03-29T04:09:55.540347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:37:51.878688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040454+0000","last_change":"2024-03-29T04:09:53.025830+0000","last_active":"2024-03-29T04:09:55.040454+0000","last_peered":"2024-03-29T04:09:55.040454+0000","last_clean":"2024-03-29T04:09:55.040454+0000","last_became_active":"2024-03-29T04:09:53.025466+0000","last_became_peered":"2024-03-29T04:09:53.025466+0000","last_unstale":"2024-03-29T04:09:55.040454+0000","last_undegraded":"2024-03-29T04:09:55.040454+0000","last_fullsized":"2024-03-29T04:09:55.040454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:36.980672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.540506+0000","last_change":"2024-03-29T04:09:54.036408+0000","last_active":"2024-03-29T04:09:55.540506+0000","last_peered":"2024-03-29T04:09:55.540506+0000","last_clean":"2024-03-29T04:09:55.540506+0000","last_became_active":"2024-03-29T04:09:54.036042+0000","last_became_peered":"2024-03-29T04:09:54.036042+0000","last_unstale":"2024-03-29T04:09:55.540506+0000","last_undegraded":"2024-03-29T04:09:55.540506+0000","last_fullsized":"2024-03-29T04:09:55.540506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:34.175737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039209+0000","last_change":"2024-03-29T04:09:53.021983+0000","last_active":"2024-03-29T04:09:55.039209+0000","last_peered":"2024-03-29T04:09:55.039209+0000","last_clean":"2024-03-29T04:09:55.039209+0000","last_became_active":"2024-03-29T04:09:53.021717+0000","last_became_peered":"2024-03-29T04:09:53.021717+0000","last_unstale":"2024-03-29T04:09:55.039209+0000","last_undegraded":"2024-03-29T04:09:55.039209+0000","last_fullsized":"2024-03-29T04:09:55.039209+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:46.362685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.037961+0000","last_change":"2024-03-29T04:09:54.030131+0000","last_active":"2024-03-29T04:09:55.037961+0000","last_peered":"2024-03-29T04:09:55.037961+0000","last_clean":"2024-03-29T04:09:55.037961+0000","last_became_active":"2024-03-29T04:09:54.029830+0000","last_became_peered":"2024-03-29T04:09:54.029830+0000","last_unstale":"2024-03-29T04:09:55.037961+0000","last_undegraded":"2024-03-29T04:09:55.037961+0000","last_fullsized":"2024-03-29T04:09:55.037961+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.103608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.040240+0000","last_change":"2024-03-29T04:09:53.022634+0000","last_active":"2024-03-29T04:09:55.040240+0000","last_peered":"2024-03-29T04:09:55.040240+0000","last_clean":"2024-03-29T04:09:55.040240+0000","last_became_active":"2024-03-29T04:09:53.022386+0000","last_became_peered":"2024-03-29T04:09:53.022386+0000","last_unstale":"2024-03-29T04:09:55.040240+0000","last_undegraded":"2024-03-29T04:09:55.040240+0000","last_fullsized":"2024-03-29T04:09:55.040240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:06.969462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039381+0000","last_change":"2024-03-29T04:09:54.028703+0000","last_active":"2024-03-29T04:09:55.039381+0000","last_peered":"2024-03-29T04:09:55.039381+0000","last_clean":"2024-03-29T04:09:55.039381+0000","last_became_active":"2024-03-29T04:09:54.028424+0000","last_became_peered":"2024-03-29T04:09:54.028424+0000","last_unstale":"2024-03-29T04:09:55.039381+0000","last_undegraded":"2024-03-29T04:09:55.039381+0000","last_fullsized":"2024-03-29T04:09:55.039381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:37.984666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038240+0000","last_change":"2024-03-29T04:09:53.017984+0000","last_active":"2024-03-29T04:09:55.038240+0000","last_peered":"2024-03-29T04:09:55.038240+0000","last_clean":"2024-03-29T04:09:55.038240+0000","last_became_active":"2024-03-29T04:09:53.017714+0000","last_became_peered":"2024-03-29T04:09:53.017714+0000","last_unstale":"2024-03-29T04:09:55.038240+0000","last_undegraded":"2024-03-29T04:09:55.038240+0000","last_fullsized":"2024-03-29T04:09:55.038240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:36.980672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.038151+0000","last_change":"2024-03-29T04:09:54.040047+0000","last_active":"2024-03-29T04:09:55.038151+0000","last_peered":"2024-03-29T04:09:55.038151+0000","last_clean":"2024-03-29T04:09:55.038151+0000","last_became_active":"2024-03-29T04:09:54.039786+0000","last_became_peered":"2024-03-29T04:09:54.039786+0000","last_unstale":"2024-03-29T04:09:55.038151+0000","last_undegraded":"2024-03-29T04:09:55.038151+0000","last_fullsized":"2024-03-29T04:09:55.038151+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:51.065761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039930+0000","last_change":"2024-03-29T04:09:53.026336+0000","last_active":"2024-03-29T04:09:55.039930+0000","last_peered":"2024-03-29T04:09:55.039930+0000","last_clean":"2024-03-29T04:09:55.039930+0000","last_became_active":"2024-03-29T04:09:53.025584+0000","last_became_peered":"2024-03-29T04:09:53.025584+0000","last_unstale":"2024-03-29T04:09:55.039930+0000","last_undegraded":"2024-03-29T04:09:55.039930+0000","last_fullsized":"2024-03-29T04:09:55.039930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:40:36.681703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018629+0000","last_change":"2024-03-29T04:09:54.028882+0000","last_active":"2024-03-29T04:09:55.018629+0000","last_peered":"2024-03-29T04:09:55.018629+0000","last_clean":"2024-03-29T04:09:55.018629+0000","last_became_active":"2024-03-29T04:09:54.028596+0000","last_became_peered":"2024-03-29T04:09:54.028596+0000","last_unstale":"2024-03-29T04:09:55.018629+0000","last_undegraded":"2024-03-29T04:09:55.018629+0000","last_fullsized":"2024-03-29T04:09:55.018629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:23.490248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023242+0000","last_change":"2024-03-29T04:09:53.015771+0000","last_active":"2024-03-29T04:09:55.023242+0000","last_peered":"2024-03-29T04:09:55.023242+0000","last_clean":"2024-03-29T04:09:55.023242+0000","last_became_active":"2024-03-29T04:09:53.015436+0000","last_became_peered":"2024-03-29T04:09:53.015436+0000","last_unstale":"2024-03-29T04:09:55.023242+0000","last_undegraded":"2024-03-29T04:09:55.023242+0000","last_fullsized":"2024-03-29T04:09:55.023242+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:14.239068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.023320+0000","last_change":"2024-03-29T04:09:54.038824+0000","last_active":"2024-03-29T04:09:55.023320+0000","last_peered":"2024-03-29T04:09:55.023320+0000","last_clean":"2024-03-29T04:09:55.023320+0000","last_became_active":"2024-03-29T04:09:54.038499+0000","last_became_peered":"2024-03-29T04:09:54.038499+0000","last_unstale":"2024-03-29T04:09:55.023320+0000","last_undegraded":"2024-03-29T04:09:55.023320+0000","last_fullsized":"2024-03-29T04:09:55.023320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:15.401317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.018636+0000","last_change":"2024-03-29T04:09:53.025251+0000","last_active":"2024-03-29T04:09:55.018636+0000","last_peered":"2024-03-29T04:09:55.018636+0000","last_clean":"2024-03-29T04:09:55.018636+0000","last_became_active":"2024-03-29T04:09:53.024927+0000","last_became_peered":"2024-03-29T04:09:53.024927+0000","last_unstale":"2024-03-29T04:09:55.018636+0000","last_undegraded":"2024-03-29T04:09:55.018636+0000","last_fullsized":"2024-03-29T04:09:55.018636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:41.222810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.021953+0000","last_change":"2024-03-29T04:09:54.037954+0000","last_active":"2024-03-29T04:09:55.021953+0000","last_peered":"2024-03-29T04:09:55.021953+0000","last_clean":"2024-03-29T04:09:55.021953+0000","last_became_active":"2024-03-29T04:09:54.037709+0000","last_became_peered":"2024-03-29T04:09:54.037709+0000","last_unstale":"2024-03-29T04:09:55.021953+0000","last_undegraded":"2024-03-29T04:09:55.021953+0000","last_fullsized":"2024-03-29T04:09:55.021953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:53.001167+0000","last_clean_scrub_stamp":"2024-03-29T04:09:53.001167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:38.793033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.039328+0000","last_change":"2024-03-29T04:09:53.026838+0000","last_active":"2024-03-29T04:09:55.039328+0000","last_peered":"2024-03-29T04:09:55.039328+0000","last_clean":"2024-03-29T04:09:55.039328+0000","last_became_active":"2024-03-29T04:09:53.026509+0000","last_became_peered":"2024-03-29T04:09:53.026509+0000","last_unstale":"2024-03-29T04:09:55.039328+0000","last_undegraded":"2024-03-29T04:09:55.039328+0000","last_fullsized":"2024-03-29T04:09:55.039328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:09:51.997173+0000","last_clean_scrub_stamp":"2024-03-29T04:09:51.997173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:27.437037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022677+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:09:55.022677+0000","last_peered":"2024-03-29T04:09:55.022677+0000","last_clean":"2024-03-29T04:09:55.022677+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:09:55.022677+0000","last_undegraded":"2024-03-29T04:09:55.022677+0000","last_fullsized":"2024-03-29T04:09:55.022677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542162,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":978944,"data_stored":740617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542162,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":151793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":143929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":151793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":740527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:09:58.389 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T04:09:58.389 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T04:09:58.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.556+0000 7f2ff311c640 1 Processor -- start 2024-03-29T04:09:58.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.556+0000 7f2ff311c640 1 -- start start 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff311c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2fec15e920 0x7f2fec0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff311c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec0cf190 con 0x7f2fec0d2120 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff311c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec0d3080 con 0x7f2fec0cfc30 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff311c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2fec0cf7a0 con 0x7f2fec15e920 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff0eb8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2fec15e920 0x7f2fec0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2febfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2fec0d2120 0x7f2fec06c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44694/0 (socket says 172.21.15.156:44694) 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2febfff640 1 -- 172.21.15.156:0/3552123184 learned_addr learned my addr 172.21.15.156:0/3552123184 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff0eb8640 1 -- 172.21.15.156:0/3552123184 >> v1:172.21.15.203:6789/0 conn(0x7f2fec0cfc30 legacy=0x7f2fec0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff0eb8640 1 -- 172.21.15.156:0/3552123184 >> v1:172.21.15.156:6789/0 conn(0x7f2fec0d2120 legacy=0x7f2fec06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2ff0eb8640 1 -- 172.21.15.156:0/3552123184 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec0cf430 con 0x7f2fec15e920 2024-03-29T04:09:58.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.557+0000 7f2feb7fe640 1 -- 172.21.15.156:0/3552123184 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1499318436 0 0) 0x7f2fec0cf190 con 0x7f2fec0d2120 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.558+0000 7f2ff0eb8640 1 --2- 172.21.15.156:0/3552123184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec15e920 0x7f2fec0d26e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0097a0 tx=0x7f2fdc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6776d0e9b541925e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2feb7fe640 1 -- 172.21.15.156:0/3552123184 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc035020 con 0x7f2fec15e920 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2feb7fe640 1 -- 172.21.15.156:0/3552123184 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc031830 con 0x7f2fec15e920 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2feb7fe640 1 -- 172.21.15.156:0/3552123184 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc031c00 con 0x7f2fec15e920 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 -- 172.21.15.156:0/3552123184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec15e920 msgr2=0x7f2fec0d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 --2- 172.21.15.156:0/3552123184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec15e920 0x7f2fec0d26e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0097a0 tx=0x7f2fdc02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 -- 172.21.15.156:0/3552123184 shutdown_connections 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 --2- 172.21.15.156:0/3552123184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec15e920 0x7f2fec0d26e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 -- 172.21.15.156:0/3552123184 >> 172.21.15.156:0/3552123184 conn(0x7f2fec09d7a0 msgr2=0x7f2fec0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:58.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 -- 172.21.15.156:0/3552123184 shutdown_connections 2024-03-29T04:09:58.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.559+0000 7f2ff311c640 1 -- 172.21.15.156:0/3552123184 wait complete. 2024-03-29T04:09:58.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.561+0000 7f2ff311c640 1 Processor -- start 2024-03-29T04:09:58.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.561+0000 7f2ff311c640 1 -- start start 2024-03-29T04:09:58.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.561+0000 7f2ff311c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fec0cfc30 0x7f2fec1ea700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.542 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 0x7f2fec1582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fec15e920 0x7f2fec158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec0d34e0 con 0x7f2fec0d2120 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec0d3360 con 0x7f2fec0cfc30 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec0d2c20 con 0x7f2fec15e920 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 0x7f2fec1582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 0x7f2fec1582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35596/0 (socket says 172.21.15.156:35596) 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 -- 172.21.15.156:0/631771670 learned_addr learned my addr 172.21.15.156:0/631771670 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff0eb8640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fec0cfc30 0x7f2fec1ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff16b9640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fec15e920 0x7f2fec158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 -- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fec15e920 msgr2=0x7f2fec158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fec15e920 0x7f2fec158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 -- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fec0cfc30 msgr2=0x7f2fec1ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fec0cfc30 0x7f2fec1ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 -- 172.21.15.156:0/631771670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec1e9660 con 0x7f2fec0d2120 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2febfff640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 0x7f2fec1582f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c500 tx=0x7f2fe000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0018070 con 0x7f2fec0d2120 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe0007580 con 0x7f2fec0d2120 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fec1e9940 con 0x7f2fec0d2120 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0004a20 con 0x7f2fec0d2120 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.562+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fec1e9f00 con 0x7f2fec0d2120 2024-03-29T04:09:58.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.563+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fec0a2480 con 0x7f2fec0d2120 2024-03-29T04:09:58.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.564+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2fe001b070 con 0x7f2fec0d2120 2024-03-29T04:09:58.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.565+0000 7f2fe97fa640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2fd806c530 0x7f2fd806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.565+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f2fe008a8f0 con 0x7f2fec0d2120 2024-03-29T04:09:58.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.565+0000 7f2ff0eb8640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2fd806c530 0x7f2fd806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.565+0000 7f2ff0eb8640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2fd806c530 0x7f2fd806e9f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc036000 tx=0x7f2fdc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.569+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2fe0058950 con 0x7f2fec0d2120 2024-03-29T04:09:58.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.819+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f2fec06e110 con 0x7f2fec0d2120 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.821+0000 7f2fe97fa640 1 -- 172.21.15.156:0/631771670 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f2fe005cc50 con 0x7f2fec0d2120 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCePgZmw7rsAxAA9fTOYwz/xRpNNRZVKRBg3w== 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stdout: caps mds = "allow" 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stdout: caps mon = "allow r" 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-29T04:09:58.801 INFO:teuthology.orchestra.run.smithi156.stderr:updated caps for client.0 2024-03-29T04:09:58.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.823+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2fd806c530 msgr2=0x7f2fd806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.823+0000 7f2ff311c640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2fd806c530 0x7f2fd806e9f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc036000 tx=0x7f2fdc03d040 comp rx=0 tx=0).stop 2024-03-29T04:09:58.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.823+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 msgr2=0x7f2fec1582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.823+0000 7f2ff311c640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 0x7f2fec1582f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c500 tx=0x7f2fe000c9c0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 shutdown_connections 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f2fd806c530 0x7f2fd806e9f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fec15e920 0x7f2fec158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fec0d2120 0x7f2fec1582f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 --2- 172.21.15.156:0/631771670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fec0cfc30 0x7f2fec1ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 >> 172.21.15.156:0/631771670 conn(0x7f2fec09d7a0 msgr2=0x7f2fec0b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 shutdown_connections 2024-03-29T04:09:58.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.824+0000 7f2ff311c640 1 -- 172.21.15.156:0/631771670 wait complete. 2024-03-29T04:09:58.815 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:09:58.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.937+0000 7fa23c493640 1 Processor -- start 2024-03-29T04:09:58.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.937+0000 7fa23c493640 1 -- start start 2024-03-29T04:09:58.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.938+0000 7fa23c493640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa23415e720 0x7fa2340b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.938+0000 7fa23c493640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa23406b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.938+0000 7fa23c493640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa23415eb00 con 0x7fa23406baf0 2024-03-29T04:09:58.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.938+0000 7fa23c493640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa2340b5b30 con 0x7fa23415e720 2024-03-29T04:09:58.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.938+0000 7fa23c493640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa2340b2bb0 con 0x7fa2340b40d0 2024-03-29T04:09:58.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.938+0000 7fa239a2e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa23406b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa23aa30640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa23406baf0 0x7fa234162240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:40952/0 (socket says 172.21.15.156:40952) 2024-03-29T04:09:58.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa23aa30640 1 -- 172.21.15.156:0/2272269215 learned_addr learned my addr 172.21.15.156:0/2272269215 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:58.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa23a22f640 1 --2- 172.21.15.156:0/2272269215 >> v2:172.21.15.203:3300/0 conn(0x7fa23415e720 0x7fa2340b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa239a2e640 1 -- 172.21.15.156:0/2272269215 >> v2:172.21.15.203:3300/0 conn(0x7fa23415e720 msgr2=0x7fa2340b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:58.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa239a2e640 1 --2- 172.21.15.156:0/2272269215 >> v2:172.21.15.203:3300/0 conn(0x7fa23415e720 0x7fa2340b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa239a2e640 1 -- 172.21.15.156:0/2272269215 >> v1:172.21.15.203:6789/0 conn(0x7fa23406baf0 legacy=0x7fa234162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa239a2e640 1 -- 172.21.15.156:0/2272269215 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2340b5d60 con 0x7fa2340b40d0 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.939+0000 7fa23a22f640 1 --2- 172.21.15.156:0/2272269215 >> v2:172.21.15.203:3300/0 conn(0x7fa23415e720 0x7fa2340b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.940+0000 7fa239a2e640 1 --2- 172.21.15.156:0/2272269215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa23406b5b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2300097a0 tx=0x7fa23002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bccd768f371282d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.940+0000 7fa23922d640 1 -- 172.21.15.156:0/2272269215 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230035020 con 0x7fa2340b40d0 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.940+0000 7fa23922d640 1 -- 172.21.15.156:0/2272269215 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa230031830 con 0x7fa2340b40d0 2024-03-29T04:09:58.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.940+0000 7fa23922d640 1 -- 172.21.15.156:0/2272269215 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230031c00 con 0x7fa2340b40d0 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 -- 172.21.15.156:0/2272269215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 msgr2=0x7fa23406b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 --2- 172.21.15.156:0/2272269215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa23406b5b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2300097a0 tx=0x7fa23002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 -- 172.21.15.156:0/2272269215 shutdown_connections 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 --2- 172.21.15.156:0/2272269215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa23406b5b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 --2- 172.21.15.156:0/2272269215 >> v2:172.21.15.203:3300/0 conn(0x7fa23415e720 0x7fa2340b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 -- 172.21.15.156:0/2272269215 >> 172.21.15.156:0/2272269215 conn(0x7fa23409aa40 msgr2=0x7fa2340b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 -- 172.21.15.156:0/2272269215 shutdown_connections 2024-03-29T04:09:58.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.941+0000 7fa23c493640 1 -- 172.21.15.156:0/2272269215 wait complete. 2024-03-29T04:09:58.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 Processor -- start 2024-03-29T04:09:58.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 -- start start 2024-03-29T04:09:58.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 0x7fa2341532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa234156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23415e720 0x7fa234148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2340d1920 con 0x7fa23406baf0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2340b6320 con 0x7fa23415e720 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23c493640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa2340d1e40 con 0x7fa2340b40d0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23a22f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 0x7fa2341532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.943+0000 7fa23a22f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 0x7fa2341532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35598/0 (socket says 172.21.15.156:35598) 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 -- 172.21.15.156:0/3871310829 learned_addr learned my addr 172.21.15.156:0/3871310829 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa239a2e640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa234156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23aa30640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23415e720 0x7fa234148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 -- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 msgr2=0x7fa234156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa234156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 -- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23415e720 msgr2=0x7fa234148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23415e720 0x7fa234148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 -- 172.21.15.156:0/3871310829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa234156aa0 con 0x7fa23406baf0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23a22f640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 0x7fa2341532d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa22400ebe0 tx=0x7fa22400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224017070 con 0x7fa23406baf0 2024-03-29T04:09:58.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa22400cc30 con 0x7fa23406baf0 2024-03-29T04:09:58.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2341599d0 con 0x7fa23406baf0 2024-03-29T04:09:58.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224013470 con 0x7fa23406baf0 2024-03-29T04:09:58.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.944+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa234159f90 con 0x7fa23406baf0 2024-03-29T04:09:58.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.945+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa234068d20 con 0x7fa23406baf0 2024-03-29T04:09:58.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.946+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa22401a070 con 0x7fa23406baf0 2024-03-29T04:09:58.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.946+0000 7fa22affd640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa21806c530 0x7fa21806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:58.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.946+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fa22408b960 con 0x7fa23406baf0 2024-03-29T04:09:58.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.946+0000 7fa239a2e640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa21806c530 0x7fa21806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:58.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.946+0000 7fa239a2e640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa21806c530 0x7fa21806e9f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa230036000 tx=0x7fa230008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:58.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:58.950+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa224059930 con 0x7fa23406baf0 2024-03-29T04:09:58.977 INFO:tasks.ceph.mds.a.smithi156.stderr:2024-03-29T04:09:58.998+0000 7f8ad5423640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T04:09:59.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.168+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa23406b8d0 con 0x7fa23406baf0 2024-03-29T04:09:59.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.169+0000 7fa22affd640 1 -- 172.21.15.156:0/3871310829 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5190 (secure 0 0 0) 0x7fa22405dc30 con 0x7fa23406baf0 2024-03-29T04:09:59.148 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:09:59.148 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:09:53.360769+0000","modified":"2024-03-29T04:09:56.506914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5310":{"gid":5310,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/788767406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":788767406},{"type":"v1","addr":"172.21.15.156:6837","nonce":788767406}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5310,"qdb_cluster":[5310]},"id":2}]} 2024-03-29T04:09:59.149 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 15 2024-03-29T04:09:59.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa21806c530 msgr2=0x7fa21806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa21806c530 0x7fa21806e9f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa230036000 tx=0x7fa230008030 comp rx=0 tx=0).stop 2024-03-29T04:09:59.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 msgr2=0x7fa2341532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 0x7fa2341532d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa22400ebe0 tx=0x7fa22400c520 comp rx=0 tx=0).stop 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 shutdown_connections 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa21806c530 0x7fa21806e9f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23415e720 0x7fa234148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2340b40d0 0x7fa234156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 --2- 172.21.15.156:0/3871310829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23406baf0 0x7fa2341532d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 >> 172.21.15.156:0/3871310829 conn(0x7fa23409aa40 msgr2=0x7fa2340aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 shutdown_connections 2024-03-29T04:09:59.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.171+0000 7fa23c493640 1 -- 172.21.15.156:0/3871310829 wait complete. 2024-03-29T04:09:59.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T04:09:53.360769+0000', 'modified': '2024-03-29T04:09:56.506914+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5310}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5310': {'gid': 5310, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6837/788767406', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6835', 'nonce': 788767406}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 788767406}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5310, 'qdb_cluster': [5310]} 2024-03-29T04:09:59.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T04:09:59.565 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T04:09:59.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.689+0000 7eff38300640 1 Processor -- start 2024-03-29T04:09:59.669 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.689+0000 7eff38300640 1 -- start start 2024-03-29T04:09:59.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff38300640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff3006ccd0 0x7eff300b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff38300640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff300cdea0 con 0x7eff301425e0 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff38300640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff300ce180 con 0x7eff300cd4a0 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff38300640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7eff3015c910 con 0x7eff3006ccd0 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff3689d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff3006ccd0 0x7eff300b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff3609c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7eff301425e0 0x7eff3015d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44706/0 (socket says 172.21.15.156:44706) 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff3609c640 1 -- 172.21.15.156:0/406647069 learned_addr learned my addr 172.21.15.156:0/406647069 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff3689d640 1 -- 172.21.15.156:0/406647069 >> v1:172.21.15.203:6789/0 conn(0x7eff300cd4a0 legacy=0x7eff3006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff3689d640 1 -- 172.21.15.156:0/406647069 >> v1:172.21.15.156:6789/0 conn(0x7eff301425e0 legacy=0x7eff3015d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.691+0000 7eff3689d640 1 -- 172.21.15.156:0/406647069 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3015cb70 con 0x7eff3006ccd0 2024-03-29T04:09:59.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.692+0000 7eff3689d640 1 --2- 172.21.15.156:0/406647069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff300b3ae0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7eff28009d90 tx=0x7eff2802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26d17054652ea3e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.692+0000 7eff3509a640 1 -- 172.21.15.156:0/406647069 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28039070 con 0x7eff3006ccd0 2024-03-29T04:09:59.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.692+0000 7eff3509a640 1 -- 172.21.15.156:0/406647069 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2802cab0 con 0x7eff3006ccd0 2024-03-29T04:09:59.672 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.692+0000 7eff3509a640 1 -- 172.21.15.156:0/406647069 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28034440 con 0x7eff3006ccd0 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.692+0000 7eff38300640 1 -- 172.21.15.156:0/406647069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 msgr2=0x7eff300b3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.692+0000 7eff38300640 1 --2- 172.21.15.156:0/406647069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff300b3ae0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7eff28009d90 tx=0x7eff2802bd80 comp rx=0 tx=0).stop 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.693+0000 7eff38300640 1 -- 172.21.15.156:0/406647069 shutdown_connections 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.693+0000 7eff38300640 1 --2- 172.21.15.156:0/406647069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff300b3ae0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.693+0000 7eff38300640 1 -- 172.21.15.156:0/406647069 >> 172.21.15.156:0/406647069 conn(0x7eff300720d0 msgr2=0x7eff300724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.693+0000 7eff38300640 1 -- 172.21.15.156:0/406647069 shutdown_connections 2024-03-29T04:09:59.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.693+0000 7eff38300640 1 -- 172.21.15.156:0/406647069 wait complete. 2024-03-29T04:09:59.675 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 Processor -- start 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 -- start start 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff3013e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff300cd4a0 0x7eff3013fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff301425e0 0x7eff301e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7eff301429c0 con 0x7eff3006ccd0 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff300cce90 con 0x7eff301425e0 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.694+0000 7eff38300640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7eff30140590 con 0x7eff300cd4a0 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff3013e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff3013e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35612/0 (socket says 172.21.15.156:35612) 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 -- 172.21.15.156:0/3942362487 learned_addr learned my addr 172.21.15.156:0/3942362487 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3689d640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff301425e0 0x7eff301e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3589b640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff300cd4a0 0x7eff3013fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 -- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff300cd4a0 msgr2=0x7eff3013fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff300cd4a0 0x7eff3013fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 -- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff301425e0 msgr2=0x7eff301e5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff301425e0 0x7eff301e5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 -- 172.21.15.156:0/3942362487 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff30140870 con 0x7eff3006ccd0 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3689d640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff301425e0 0x7eff301e5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff3609c640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff3013e1a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00c5f0 tx=0x7eff2c00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff26ffd640 1 -- 172.21.15.156:0/3942362487 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c018070 con 0x7eff3006ccd0 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff26ffd640 1 -- 172.21.15.156:0/3942362487 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2c0074e0 con 0x7eff3006ccd0 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff30140b50 con 0x7eff3006ccd0 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff26ffd640 1 -- 172.21.15.156:0/3942362487 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c004a00 con 0x7eff3006ccd0 2024-03-29T04:09:59.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.695+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3013e6e0 con 0x7eff3006ccd0 2024-03-29T04:09:59.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.697+0000 7eff26ffd640 1 -- 172.21.15.156:0/3942362487 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7eff2c01b070 con 0x7eff3006ccd0 2024-03-29T04:09:59.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.697+0000 7eff26ffd640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff1406c560 0x7eff1406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.698+0000 7eff26ffd640 1 -- 172.21.15.156:0/3942362487 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7eff2c08ab30 con 0x7eff3006ccd0 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.698+0000 7eff3589b640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff1406c560 0x7eff1406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.698+0000 7eff3589b640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff1406c560 0x7eff1406ea20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7eff30160ec0 tx=0x7eff20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.698+0000 7eff3afc2740 1 Processor -- start 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 -- start start 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d5eb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d61b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d5ebd40 0x56193d61bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56193d555020 con 0x56193d618c40 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56193d4fd5f0 con 0x56193d5ebd40 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x56193d550fd0 con 0x56193d618860 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3589b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d61b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3589b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d61b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35626/0 (socket says 172.21.15.156:35626) 2024-03-29T04:09:59.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.699+0000 7eff3589b640 1 -- 172.21.15.156:0/909483484 learned_addr learned my addr 172.21.15.156:0/909483484 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3609c640 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d5eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3689d640 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d5ebd40 0x56193d61bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3589b640 1 -- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 msgr2=0x56193d5eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3589b640 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d5eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3589b640 1 -- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d5ebd40 msgr2=0x56193d61bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3589b640 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d5ebd40 0x56193d61bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3589b640 1 -- 172.21.15.156:0/909483484 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56193d2692d0 con 0x56193d618c40 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff3589b640 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d61b5f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7eff200052f0 tx=0x7eff20011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbbcceaa8263e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff13fff640 1 -- 172.21.15.156:0/909483484 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2000e530 con 0x56193d618c40 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff13fff640 1 -- 172.21.15.156:0/909483484 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2000e6d0 con 0x56193d618c40 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.700+0000 7eff13fff640 1 -- 172.21.15.156:0/909483484 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2000ea80 con 0x56193d618c40 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 -- 172.21.15.156:0/909483484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 msgr2=0x56193d61b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d61b5f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7eff200052f0 tx=0x7eff20011000 comp rx=0 tx=0).stop 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 -- 172.21.15.156:0/909483484 shutdown_connections 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d5ebd40 0x56193d61bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d61b5f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 --2- 172.21.15.156:0/909483484 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d5eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 -- 172.21.15.156:0/909483484 >> 172.21.15.156:0/909483484 conn(0x56193d5eda70 msgr2=0x56193d6365f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 -- 172.21.15.156:0/909483484 shutdown_connections 2024-03-29T04:09:59.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.701+0000 7eff3afc2740 1 -- 172.21.15.156:0/909483484 wait complete. 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.702+0000 7eff3afc2740 1 -- start start 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.702+0000 7eff3afc2740 1 Processor -- start 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d61d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d618c40 0x56193d64a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 0x56193d64a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56193d5f0110 con 0x56193d6467f0 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56193d5109c0 con 0x56193d618c40 2024-03-29T04:09:59.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x56193d5f2950 con 0x56193d618860 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3afc2740 10 client.0 _ll_register_callbacks cb 0x56193d4f4fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3689d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 0x56193d64a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3689d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 0x56193d64a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35634/0 (socket says 172.21.15.156:35634) 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3689d640 1 -- 172.21.15.156:0/3060699819 learned_addr learned my addr 172.21.15.156:0/3060699819 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3589b640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d618c40 0x56193d64a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.703+0000 7eff3609c640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d61d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3689d640 1 -- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 msgr2=0x56193d61d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3689d640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d61d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3689d640 1 -- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d618c40 msgr2=0x56193d64a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3689d640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d618c40 0x56193d64a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3689d640 1 -- 172.21.15.156:0/3060699819 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56193d3336b0 con 0x56193d6467f0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3689d640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 0x56193d64a690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7eff2802bdc0 tx=0x7eff28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49e38c9acce4d34 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff117fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:09:59.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff117fa640 1 -- 172.21.15.156:0/3060699819 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28039040 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff117fa640 1 -- 172.21.15.156:0/3060699819 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7eff28034bf0 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff2000e6d0 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff117fa640 1 -- 172.21.15.156:0/3060699819 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7eff28051050 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 10 client.5499 fetch_fsmap learned FSMap version 15 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x56193d61e3f0 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff117fa640 1 -- 172.21.15.156:0/3060699819 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2000e6d0 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff117fa640 1 -- 172.21.15.156:0/3060699819 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7eff28054020 con 0x56193d6467f0 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 10 client.5499 fetch_fsmap finished waiting for FSMap version 15 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 10 client.5499 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 20 client.5499 populate_metadata read hostname 'smithi156' 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d605740 0x56193d608a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.704+0000 7eff3afc2740 4 client.5499 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.705+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56193d5294e0 con 0x56193d605740 2024-03-29T04:09:59.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.705+0000 7eff3609c640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d605740 0x56193d608a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.705+0000 7eff117fa640 10 client.5499 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.705+0000 7eff3609c640 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d605740 0x56193d608a50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11d1fee09c20e253 server_cookie=a91882ebec1756dc in_seq=0 out_seq=0 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff117fa640 1 -- 172.21.15.156:0/3060699819 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23239 (crc 0 0 0) 0x56193d5294e0 con 0x56193d605740 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff117fa640 10 client.5499 handle_command_reply: tid=0 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 client.5499 shutdown 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 msgr2=0x56193d64a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 0x56193d64a690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7eff2802bdc0 tx=0x7eff28004ab0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 shutdown_connections 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d605740 0x56193d608a50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d6467f0 0x56193d64a690 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d618c40 0x56193d64a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff117fa640 0 client.5499 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3060699819 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d61d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 >> 172.21.15.156:0/3060699819 conn(0x56193d5ebd40 msgr2=0x56193d5ef8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 shutdown_connections 2024-03-29T04:09:59.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 1 -- 172.21.15.156:0/3060699819 wait complete. 2024-03-29T04:09:59.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.706+0000 7eff3afc2740 20 client.5499 trim_cache size 0 max 16384 2024-03-29T04:09:59.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.724+0000 7eff3afc2740 1 Processor -- start 2024-03-29T04:09:59.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.724+0000 7eff3afc2740 1 -- start start 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d6515e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d6397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d639d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3afc2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7eff301fdae0 con 0x56193d618c40 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff300aa180 con 0x56193d6467f0 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7eff300b48a0 con 0x56193d618860 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d6397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d6397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35636/0 (socket says 172.21.15.156:35636) 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 -- 172.21.15.156:0/2838375651 learned_addr learned my addr 172.21.15.156:0/2838375651 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 -- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 msgr2=0x56193d6515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3609c640 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d6515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3689d640 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d639d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d6515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 -- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 msgr2=0x56193d639d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d639d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 -- 172.21.15.156:0/2838375651 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56193d607c80 con 0x56193d618c40 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3689d640 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d639d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.725+0000 7eff3589b640 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d6397c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7eff20014000 tx=0x7eff20005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950b8661fb1b62c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff137fe640 1 -- 172.21.15.156:0/2838375651 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2000e9c0 con 0x56193d618c40 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff137fe640 1 -- 172.21.15.156:0/2838375651 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2000eb60 con 0x56193d618c40 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff137fe640 1 -- 172.21.15.156:0/2838375651 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff20021450 con 0x56193d618c40 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 -- 172.21.15.156:0/2838375651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 msgr2=0x56193d6397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d6397c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7eff20014000 tx=0x7eff20005000 comp rx=0 tx=0).stop 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 -- 172.21.15.156:0/2838375651 shutdown_connections 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d639d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618c40 0x56193d6397c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.713 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 --2- 172.21.15.156:0/2838375651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618860 0x56193d6515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 -- 172.21.15.156:0/2838375651 >> 172.21.15.156:0/2838375651 conn(0x56193d5ebd40 msgr2=0x56193d606a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 -- 172.21.15.156:0/2838375651 shutdown_connections 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.726+0000 7eff3afc2740 1 -- 172.21.15.156:0/2838375651 wait complete. 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.727+0000 7eff3afc2740 1 -- start start 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.727+0000 7eff3afc2740 1 Processor -- start 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 0x56193d6494d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618c40 0x56193d649a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d6b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56193d6055c0 con 0x56193d618860 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56193d6051c0 con 0x56193d6467f0 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x56193d605c10 con 0x56193d618c40 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 10 client.0 _ll_register_callbacks cb 0x7eff28037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:09:59.714 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 0x56193d6494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 0x56193d6494d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35638/0 (socket says 172.21.15.156:35638) 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 -- 172.21.15.156:0/3807878374 learned_addr learned my addr 172.21.15.156:0/3807878374 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3589b640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618c40 0x56193d649a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3689d640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d6b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 -- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618c40 msgr2=0x56193d649a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618c40 0x56193d649a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 -- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 msgr2=0x56193d6b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d6b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 -- 172.21.15.156:0/3807878374 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56193d5eb3e0 con 0x56193d618860 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3589b640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618c40 0x56193d649a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3609c640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 0x56193d6494d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7eff2c058750 tx=0x7eff2c05b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f13e49369bf3db server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff117fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff117fa640 1 -- 172.21.15.156:0/3807878374 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c061630 con 0x56193d618860 2024-03-29T04:09:59.715 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff117fa640 1 -- 172.21.15.156:0/3807878374 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7eff2c0550c0 con 0x56193d618860 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.728+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff2000eb60 con 0x56193d618860 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff117fa640 1 -- 172.21.15.156:0/3807878374 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7eff2c004ba0 con 0x56193d618860 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 10 client.5505 fetch_fsmap learned FSMap version 15 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x56193d61b980 con 0x56193d618860 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff117fa640 1 -- 172.21.15.156:0/3807878374 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2000eb60 con 0x56193d618860 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff117fa640 1 -- 172.21.15.156:0/3807878374 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7eff2c05f070 con 0x56193d618860 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 10 client.5505 fetch_fsmap finished waiting for FSMap version 15 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 10 client.5505 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 20 client.5505 populate_metadata read hostname 'smithi156' 2024-03-29T04:09:59.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d60aa60 0x56193d60ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 4 client.5505 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "status"}) -- 0x56193d562560 con 0x56193d60aa60 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3589b640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d60aa60 0x56193d60ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff117fa640 10 client.5505 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3589b640 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d60aa60 0x56193d60ae40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e83e228d6da825c3 server_cookie=27154506ae78f2c4 in_seq=0 out_seq=0 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff117fa640 1 -- 172.21.15.156:0/3807878374 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56193d562560 con 0x56193d60aa60 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff117fa640 10 client.5505 handle_command_reply: tid=0 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.729+0000 7eff3afc2740 1 client.5505 shutdown 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 msgr2=0x56193d6494d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 0x56193d6494d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7eff2c058750 tx=0x7eff2c05b040 comp rx=0 tx=0).stop 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 shutdown_connections 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x56193d60aa60 0x56193d60ae40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56193d6467f0 0x56193d6b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.717 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56193d618c40 0x56193d649a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff117fa640 0 client.5505 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 --2- 172.21.15.156:0/3807878374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56193d618860 0x56193d6494d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 >> 172.21.15.156:0/3807878374 conn(0x56193d5ebd40 msgr2=0x56193d6397c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 shutdown_connections 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 1 -- 172.21.15.156:0/3807878374 wait complete. 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff3afc2740 20 client.5505 trim_cache size 0 max 16384 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "6a348a99-5402-4006-8e25-31481aaa0bf5", 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 0, 2024-03-29T04:09:59.718 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 5310, 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 5.6997541490000003, 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 15, 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 34, 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 34, 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 12.585704959999999 2024-03-29T04:09:59.719 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:09:59.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff1406c560 msgr2=0x7eff1406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff1406c560 0x7eff1406ea20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7eff30160ec0 tx=0x7eff20009040 comp rx=0 tx=0).stop 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 msgr2=0x7eff3013e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff3013e1a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00c5f0 tx=0x7eff2c00cab0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 shutdown_connections 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff1406c560 0x7eff1406ea20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff301425e0 0x7eff301e5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff300cd4a0 0x7eff3013fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 --2- 172.21.15.156:0/3942362487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3006ccd0 0x7eff3013e1a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 >> 172.21.15.156:0/3942362487 conn(0x7eff300720d0 msgr2=0x7eff3015f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 shutdown_connections 2024-03-29T04:09:59.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.730+0000 7eff38300640 1 -- 172.21.15.156:0/3942362487 wait complete. 2024-03-29T04:09:59.722 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T04:09:59.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.844+0000 7f7cc3d8d640 1 Processor -- start 2024-03-29T04:09:59.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.845+0000 7f7cc3d8d640 1 -- start start 2024-03-29T04:09:59.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.845+0000 7f7cc3d8d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc15e730 0x7f7cbc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.845+0000 7f7cc3d8d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cbc0d49d0 0x7f7cbc06cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.845+0000 7f7cc3d8d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cbc0cfa20 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.845+0000 7f7cc3d8d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7cbc06c720 con 0x7f7cbc0d49d0 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.845+0000 7f7cc3d8d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc0cfd00 con 0x7f7cbc15e730 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cbc0d49d0 0x7f7cbc06cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cbc0d49d0 0x7f7cbc06cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35652/0 (socket says 172.21.15.156:35652) 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 -- 172.21.15.156:0/2554375192 learned_addr learned my addr 172.21.15.156:0/2554375192 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc1b29640 1 --2- 172.21.15.156:0/2554375192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc15e730 0x7f7cbc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 -- 172.21.15.156:0/2554375192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc15e730 msgr2=0x7f7cbc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 --2- 172.21.15.156:0/2554375192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc15e730 0x7f7cbc0ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 -- 172.21.15.156:0/2554375192 >> v1:172.21.15.203:6789/0 conn(0x7f7cbc0cf3e0 legacy=0x7f7cbc0d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:09:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.846+0000 7f7cc232a640 1 -- 172.21.15.156:0/2554375192 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc0cffe0 con 0x7f7cbc0d49d0 2024-03-29T04:09:59.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc232a640 1 --2- 172.21.15.156:0/2554375192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0d49d0 0x7f7cbc06cbb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009e50 tx=0x7f7cb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee7894d2a3329ea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc0b27640 1 -- 172.21.15.156:0/2554375192 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb402ca70 con 0x7f7cbc0d49d0 2024-03-29T04:09:59.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc0b27640 1 -- 172.21.15.156:0/2554375192 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb402cc10 con 0x7f7cbc0d49d0 2024-03-29T04:09:59.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc0b27640 1 -- 172.21.15.156:0/2554375192 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40343f0 con 0x7f7cbc0d49d0 2024-03-29T04:09:59.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/2554375192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0d49d0 msgr2=0x7f7cbc06cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/2554375192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0d49d0 0x7f7cbc06cbb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009e50 tx=0x7f7cb402bed0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/2554375192 shutdown_connections 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/2554375192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0d49d0 0x7f7cbc06cbb0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/2554375192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc15e730 0x7f7cbc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/2554375192 >> 172.21.15.156:0/2554375192 conn(0x7f7cbc08bfd0 msgr2=0x7f7cbc09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/2554375192 shutdown_connections 2024-03-29T04:09:59.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.847+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/2554375192 wait complete. 2024-03-29T04:09:59.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.848+0000 7f7cc3d8d640 1 Processor -- start 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 -- start start 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 0x7f7cbc0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc0d49d0 0x7f7cbc0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cbc15e730 0x7f7cbc15f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc15eb10 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0d2220 con 0x7f7cbc15e730 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc3d8d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc15ec90 con 0x7f7cbc0d49d0 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc1b29640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 0x7f7cbc0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc1b29640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 0x7f7cbc0cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35654/0 (socket says 172.21.15.156:35654) 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.849+0000 7f7cc1b29640 1 -- 172.21.15.156:0/4015311393 learned_addr learned my addr 172.21.15.156:0/4015311393 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1328640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc0d49d0 0x7f7cbc0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc232a640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cbc15e730 0x7f7cbc15f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1b29640 1 -- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc0d49d0 msgr2=0x7f7cbc0c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1b29640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc0d49d0 0x7f7cbc0c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1b29640 1 -- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cbc15e730 msgr2=0x7f7cbc15f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1b29640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cbc15e730 0x7f7cbc15f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1b29640 1 -- 172.21.15.156:0/4015311393 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc0c7fd0 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1328640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc0d49d0 0x7f7cbc0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc232a640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cbc15e730 0x7f7cbc15f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc1b29640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 0x7f7cbc0cacf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800c7f0 tx=0x7f7cb800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8015070 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb8004220 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cbc0c82b0 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8003610 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.850+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cbc0c8870 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.851+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cbc068610 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.852+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7cb80039c0 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.852+0000 7f7cb27fc640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ca006c560 0x7f7ca006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:09:59.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.852+0000 7f7cc1328640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ca006c560 0x7f7ca006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:09:59.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.852+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f7cb808b6b0 con 0x7f7cbc0cf3e0 2024-03-29T04:09:59.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.853+0000 7f7cc1328640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ca006c560 0x7f7ca006ea20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0ce3d0 tx=0x7f7cac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:09:59.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:09:59.857+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7cb8059cd0 con 0x7f7cbc0cf3e0 2024-03-29T04:10:00.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.076+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7cbc06db10 con 0x7f7cbc0cf3e0 2024-03-29T04:10:00.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.077+0000 7f7cb27fc640 1 -- 172.21.15.156:0/4015311393 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f7cb80595d0 con 0x7f7cbc0cf3e0 2024-03-29T04:10:00.056 INFO:teuthology.orchestra.run.smithi156.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T04:10:00.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ca006c560 msgr2=0x7f7ca006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:00.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ca006c560 0x7f7ca006ea20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0ce3d0 tx=0x7f7cac009040 comp rx=0 tx=0).stop 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 msgr2=0x7f7cbc0cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 0x7f7cbc0cacf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800c7f0 tx=0x7f7cb800ccb0 comp rx=0 tx=0).stop 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 shutdown_connections 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7ca006c560 0x7f7ca006ea20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cbc15e730 0x7f7cbc15f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7cbc0d49d0 0x7f7cbc0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 --2- 172.21.15.156:0/4015311393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cbc0cf3e0 0x7f7cbc0cacf0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 >> 172.21.15.156:0/4015311393 conn(0x7f7cbc08bfd0 msgr2=0x7f7cbc0b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 shutdown_connections 2024-03-29T04:10:00.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00.079+0000 7f7cc3d8d640 1 -- 172.21.15.156:0/4015311393 wait complete. 2024-03-29T04:10:00.069 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T04:10:00.069 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T04:10:00.069 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-03-29T04:10:00.070 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T04:10:00.070 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:00.070 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T04:10:00.070 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T04:10:00.070 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T04:10:00.070 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T04:10:00.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.070 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-03-29T04:10:00.087 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:10:00.087 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:10:00.087 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:10:00.087 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.087 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-03-29T04:10:00.088 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.088 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:10:00.088 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:00.088 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-03-29T04:10:00.088 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:10:00.088 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42690sec preferred_lft 42690sec 2024-03-29T04:10:00.089 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0e:08:18:d1:6a:54 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::6075:d6ff:fee6:8b38/64 scope link 2024-03-29T04:10:00.090 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.091 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T04:10:00.091 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 32:dc:33:e3:68:e4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:10:00.091 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::30dc:33ff:fee3:68e4/64 scope link 2024-03-29T04:10:00.091 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.091 INFO:teuthology.orchestra.run.smithi156.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T04:10:00.091 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0e:08:18:d1:6a:54 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:10:00.092 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::c08:18ff:fed1:6a54/64 scope link 2024-03-29T04:10:00.092 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:00.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.092 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T04:10:00.153 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-03-29T04:10:00.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.155 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-03-29T04:10:00.212 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-03-29T04:10:00.212 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T04:10:00.212 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T04:10:00.212 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T04:10:00.213 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T04:10:00.213 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:10:00.213 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-03-29T04:10:00.213 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T04:10:00.213 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T04:10:00.213 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T04:10:00.213 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-03-29T04:10:00.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:10:00.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:10:00.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.349 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-29T04:10:00.364 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T04:10:00.364 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T04:10:00.364 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T04:10:00.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.365 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2024-03-29T04:10:00.431 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-29T04:10:00.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:10:00.431 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:00.486 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T04:10:00.486 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:10:00.486 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:00.541 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-29T04:10:00.609 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:10:00.665 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-29T04:10:00.666 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-0 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-1 2024-03-29T04:10:00.667 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-2 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-3 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:10:00.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.668 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-29T04:10:00.720 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-29T04:10:00.721 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-29T04:10:00.764 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-03-29T04:10:00.791 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-03-29T04:10:00.847 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-03-29T04:10:00.848 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-0 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-1 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-2 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-3 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:10:00.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:00.849 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-03-29T04:10:00.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:2024-03-29T04:10:00.874+0000 7f5a08e0c480 -1 init, newargv = 0x55d9853dda50 newargc=15 2024-03-29T04:10:00.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:ceph-fuse[51774]: starting ceph client 2024-03-29T04:10:00.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:ceph-fuse[51774]: starting fuse 2024-03-29T04:10:00.869 INFO:teuthology.orchestra.run.smithi156.stdout:41 2024-03-29T04:10:00.869 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-03-29T04:10:00.870 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.0")) 2024-03-29T04:10:00.871 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-03-29T04:10:00.994 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.0.51774.asok 2024-03-29T04:10:00.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:10:01.006 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51774.asok 2024-03-29T04:10:01.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51774.asok status 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "69e2cd73c52284dfd5096c66b60c6e6b26a3c447", 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2520-g69e2cd73 (69e2cd73c52284dfd5096c66b60c6e6b26a3c447) squid (dev)", 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "0", 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "51774", 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-03-29T04:10:01.143 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 5514, 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 5514 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 1157355173 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-03-29T04:10:01.144 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 1157355173 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.5514 192.168.0.1:0/1157355173", 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/1157355173", 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 15, 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 34, 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-03-29T04:10:01.145 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:10:01.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.151 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-03-29T04:10:01.152 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:01.168 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-03-29T04:10:01.168 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:01.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.168 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:01.237 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T04:10:01.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.400+0000 7fb719e93640 1 Processor -- start 2024-03-29T04:10:01.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.400+0000 7fb719e93640 1 -- start start 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb719e93640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb71415fb60 0x7fb71406c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb719e93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7140cf7a0 con 0x7fb7140afdf0 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb719e93640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7140b01d0 con 0x7fb71406cb00 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb719e93640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb7140cfa10 con 0x7fb71415fb60 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb712ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb71415fb60 0x7fb71406c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb7137fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb7140afdf0 0x7fb71415c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44712/0 (socket says 172.21.15.156:44712) 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.401+0000 7fb7137fe640 1 -- 172.21.15.156:0/577910805 learned_addr learned my addr 172.21.15.156:0/577910805 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:01.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.402+0000 7fb712ffd640 1 -- 172.21.15.156:0/577910805 >> v1:172.21.15.203:6789/0 conn(0x7fb71406cb00 legacy=0x7fb7140cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.402+0000 7fb712ffd640 1 -- 172.21.15.156:0/577910805 >> v1:172.21.15.156:6789/0 conn(0x7fb7140afdf0 legacy=0x7fb71415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.402+0000 7fb712ffd640 1 -- 172.21.15.156:0/577910805 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7140cfcf0 con 0x7fb71415fb60 2024-03-29T04:10:01.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.402+0000 7fb7127fc640 1 -- 172.21.15.156:0/577910805 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148012775 0 0) 0x7fb7140cf7a0 con 0x7fb7140afdf0 2024-03-29T04:10:01.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.402+0000 7fb712ffd640 1 --2- 172.21.15.156:0/577910805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71415fb60 0x7fb71406c5c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb7080097d0 tx=0x7fb70802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b7b6e231b49742 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:01.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb7127fc640 1 -- 172.21.15.156:0/577910805 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7080315a0 con 0x7fb71415fb60 2024-03-29T04:10:01.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb7127fc640 1 -- 172.21.15.156:0/577910805 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb708037070 con 0x7fb71415fb60 2024-03-29T04:10:01.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb7127fc640 1 -- 172.21.15.156:0/577910805 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708031960 con 0x7fb71415fb60 2024-03-29T04:10:01.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 -- 172.21.15.156:0/577910805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71415fb60 msgr2=0x7fb71406c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 --2- 172.21.15.156:0/577910805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71415fb60 0x7fb71406c5c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb7080097d0 tx=0x7fb70802b840 comp rx=0 tx=0).stop 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 -- 172.21.15.156:0/577910805 shutdown_connections 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 --2- 172.21.15.156:0/577910805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71415fb60 0x7fb71406c5c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 -- 172.21.15.156:0/577910805 >> 172.21.15.156:0/577910805 conn(0x7fb714071f80 msgr2=0x7fb714072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 -- 172.21.15.156:0/577910805 shutdown_connections 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.403+0000 7fb719e93640 1 -- 172.21.15.156:0/577910805 wait complete. 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.404+0000 7fb719e93640 1 Processor -- start 2024-03-29T04:10:01.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.405+0000 7fb719e93640 1 -- start start 2024-03-29T04:10:01.386 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb719e93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 0x7fb7141521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb719e93640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb7140afdf0 0x7fb7141540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb719e93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714154630 0x7fb714155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb719e93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb71415c740 con 0x7fb71406cb00 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb719e93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb7140d02b0 con 0x7fb714154630 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb719e93640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb7140cd3e0 con 0x7fb7140afdf0 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 0x7fb7141521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 0x7fb7141521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35680/0 (socket says 172.21.15.156:35680) 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 -- 172.21.15.156:0/502488443 learned_addr learned my addr 172.21.15.156:0/502488443 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb712ffd640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb7140afdf0 0x7fb7141540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb713fff640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714154630 0x7fb714155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 -- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb7140afdf0 msgr2=0x7fb7141540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb7140afdf0 0x7fb7141540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 -- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714154630 msgr2=0x7fb714155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714154630 0x7fb714155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 -- 172.21.15.156:0/502488443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb714156c70 con 0x7fb71406cb00 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb7137fe640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 0x7fb7141521e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb70400c4a0 tx=0x7fb70400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.406+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704018070 con 0x7fb71406cb00 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.407+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7040074e0 con 0x7fb71406cb00 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.407+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb714156f50 con 0x7fb71406cb00 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.407+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704004a20 con 0x7fb71406cb00 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.407+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb714157510 con 0x7fb71406cb00 2024-03-29T04:10:01.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.408+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb714154a40 con 0x7fb71406cb00 2024-03-29T04:10:01.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.408+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb704004bc0 con 0x7fb71406cb00 2024-03-29T04:10:01.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.409+0000 7fb6fbfff640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb70006c560 0x7fb70006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:01.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.409+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fb70408a980 con 0x7fb71406cb00 2024-03-29T04:10:01.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.409+0000 7fb712ffd640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb70006c560 0x7fb70006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:01.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.409+0000 7fb712ffd640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb70006c560 0x7fb70006ea20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb708036000 tx=0x7fb70803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:01.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.413+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb714154a40 con 0x7fb71406cb00 2024-03-29T04:10:01.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.638+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb714068d20 con 0x7fb71406cb00 2024-03-29T04:10:01.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.638+0000 7fb6fbfff640 1 -- 172.21.15.156:0/502488443 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7fb704058f20 con 0x7fb71406cb00 2024-03-29T04:10:01.618 INFO:teuthology.orchestra.run.smithi156.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T04:10:01.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb70006c560 msgr2=0x7fb70006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb70006c560 0x7fb70006ea20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb708036000 tx=0x7fb70803d040 comp rx=0 tx=0).stop 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 msgr2=0x7fb7141521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 0x7fb7141521e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb70400c4a0 tx=0x7fb70400c960 comp rx=0 tx=0).stop 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 shutdown_connections 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb70006c560 0x7fb70006ea20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714154630 0x7fb714155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb7140afdf0 0x7fb7141540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 --2- 172.21.15.156:0/502488443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb71406cb00 0x7fb7141521e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 >> 172.21.15.156:0/502488443 conn(0x7fb714071f80 msgr2=0x7fb7140cd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 shutdown_connections 2024-03-29T04:10:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:01.640+0000 7fb719e93640 1 -- 172.21.15.156:0/502488443 wait complete. 2024-03-29T04:10:01.630 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi203.front.sepia.ceph.com 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T04:10:01.631 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-29T04:10:01.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.632 DEBUG:teuthology.orchestra.run.smithi203:> ip addr 2024-03-29T04:10:01.651 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T04:10:01.651 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T04:10:01.651 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T04:10:01.651 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.651 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:44 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f0 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:45 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f1 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:22 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:23 brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: inet 172.21.15.203/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft 42686sec preferred_lft 42686sec 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::ec4:7aff:fe8f:4a23/64 scope link 2024-03-29T04:10:01.652 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 5a:fe:1c:ce:ed:8c brd ff:ff:ff:ff:ff:ff 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::b881:18ff:fe2d:cce1/64 scope link 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 8a:cf:e6:65:5e:24 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::88cf:e6ff:fe65:5e24/64 scope link 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 5a:fe:1c:ce:ed:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::58fe:1cff:fece:ed8c/64 scope link 2024-03-29T04:10:01.653 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-03-29T04:10:01.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.654 DEBUG:teuthology.orchestra.run.smithi203:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T04:10:01.723 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-29T04:10:01.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.725 DEBUG:teuthology.orchestra.run.smithi203:> route 2024-03-29T04:10:01.785 INFO:teuthology.orchestra.run.smithi203.stdout:Kernel IP routing table 2024-03-29T04:10:01.785 INFO:teuthology.orchestra.run.smithi203.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T04:10:01.785 INFO:teuthology.orchestra.run.smithi203.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T04:10:01.785 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T04:10:01.786 INFO:teuthology.orchestra.run.smithi203.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T04:10:01.786 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-29T04:10:01.786 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-03-29T04:10:01.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T04:10:01.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T04:10:01.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T04:10:01.787 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-03-29T04:10:01.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:10:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-29T04:10:01.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:10:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:10:01.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.927 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-03-29T04:10:01.941 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-29T04:10:01.941 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-29T04:10:01.941 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-29T04:10:01.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:01.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set brx.0 up 2024-03-29T04:10:02.010 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-29T04:10:02.010 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:10:02.010 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:02.066 INFO:teuthology.orchestra.run.smithi203.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-29T04:10:02.066 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T04:10:02.067 DEBUG:teuthology.orchestra.run.smithi203:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:02.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-03-29T04:10:02.190 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:10:02.245 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-03-29T04:10:02.246 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2024-03-29T04:10:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:10:02.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:02.248 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-03-29T04:10:02.300 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-29T04:10:02.301 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-29T04:10:02.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-03-29T04:10:02.370 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-03-29T04:10:02.426 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-03-29T04:10:02.427 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:10:02.428 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:10:02.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:02.429 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-03-29T04:10:02.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:2024-03-29T04:10:02.450+0000 7f0255e45480 -1 init, newargv = 0x5567f708a9d0 newargc=15 2024-03-29T04:10:02.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[44721]: starting ceph client 2024-03-29T04:10:02.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[44721]: starting fuse 2024-03-29T04:10:02.445 INFO:teuthology.orchestra.run.smithi203.stdout:41 2024-03-29T04:10:02.445 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-29T04:10:02.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo stdin-killer -- python3 -c ' 2024-03-29T04:10:02.445 DEBUG:teuthology.orchestra.run.smithi203:> import glob 2024-03-29T04:10:02.445 DEBUG:teuthology.orchestra.run.smithi203:> import re 2024-03-29T04:10:02.445 DEBUG:teuthology.orchestra.run.smithi203:> import os 2024-03-29T04:10:02.445 DEBUG:teuthology.orchestra.run.smithi203:> import subprocess 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> def _find_admin_socket(client_name): 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> files = glob.glob(asok_path) 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> # Given a non-glob path, it better be there 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> if "*" not in asok_path: 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> assert(len(files) == 1) 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> return files[0] 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> for f in files: 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-29T04:10:02.446 DEBUG:teuthology.orchestra.run.smithi203:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> contents = proc_f.read() 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> if mountpoint in contents: 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> return f 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> print(_find_admin_socket("client.1")) 2024-03-29T04:10:02.447 DEBUG:teuthology.orchestra.run.smithi203:> ' 2024-03-29T04:10:02.565 INFO:teuthology.orchestra.run.smithi203.stdout:/var/run/ceph/ceph-client.1.44721.asok 2024-03-29T04:10:02.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T04:10:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-29T04:10:02.575 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44721.asok 2024-03-29T04:10:02.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:02.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44721.asok status 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "metadata": { 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_sha1": "69e2cd73c52284dfd5096c66b60c6e6b26a3c447", 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2520-g69e2cd73 (69e2cd73c52284dfd5096c66b60c6e6b26a3c447) squid (dev)", 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "entity_id": "1", 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "hostname": "smithi203", 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "pid": "44721", 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "root": "/" 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_count": 0, 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_pinned_count": 0, 2024-03-29T04:10:02.712 INFO:teuthology.orchestra.run.smithi203.stdout: "id": 5158, 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "inst": { 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "name": { 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "client", 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "num": 5158 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 2546002022 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-03-29T04:10:02.713 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 2546002022 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "inst_str": "client.5158 192.168.0.1:0/2546002022", 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "addr_str": "192.168.0.1:0/2546002022", 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "inode_count": 1, 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_epoch": 15, 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch": 34, 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch_barrier": 0, 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "blocklisted": false, 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout: "fs_name": "cephfs" 2024-03-29T04:10:02.714 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-29T04:10:02.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:02.721 DEBUG:teuthology.orchestra.run.smithi203:> stat --file-system '--printf=%T 2024-03-29T04:10:02.721 DEBUG:teuthology.orchestra.run.smithi203:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:02.736 INFO:teuthology.orchestra.run.smithi203.stdout:fuseblk 2024-03-29T04:10:02.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:02.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:02.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:02.804 DEBUG:tasks.cephfs.test_uninlining:setting inline_data:1 2024-03-29T04:10:02.804 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs inline_data 1 --yes-i-really-really-mean-it 2024-03-29T04:10:02.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.935+0000 7f530de0d640 1 Processor -- start 2024-03-29T04:10:02.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.935+0000 7f530de0d640 1 -- start start 2024-03-29T04:10:02.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f530de0d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f53080b3be0 0x7f530806c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:02.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f530de0d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f530806cb00 0x7f530815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:02.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f530de0d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5308156da0 con 0x7f53081569c0 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f530de0d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f53080b5f10 con 0x7f530806cb00 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f530de0d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f53080b4080 con 0x7f53080b3be0 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5307fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f530806cb00 0x7f530815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f53077fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f53081569c0 0x7f530815d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54306/0 (socket says 172.21.15.156:54306) 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f53077fe640 1 -- 172.21.15.156:0/1282109475 learned_addr learned my addr 172.21.15.156:0/1282109475 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5306ffd640 1 --2- 172.21.15.156:0/1282109475 >> v2:172.21.15.203:3300/0 conn(0x7f53080b3be0 0x7f530806c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5307fff640 1 -- 172.21.15.156:0/1282109475 >> v2:172.21.15.203:3300/0 conn(0x7f53080b3be0 msgr2=0x7f530806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5307fff640 1 --2- 172.21.15.156:0/1282109475 >> v2:172.21.15.203:3300/0 conn(0x7f53080b3be0 0x7f530806c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5307fff640 1 -- 172.21.15.156:0/1282109475 >> v1:172.21.15.156:6789/0 conn(0x7f53081569c0 legacy=0x7f530815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5307fff640 1 -- 172.21.15.156:0/1282109475 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53080b4360 con 0x7f530806cb00 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f53067fc640 1 -- 172.21.15.156:0/1282109475 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28339436 0 0) 0x7f5308156da0 con 0x7f53081569c0 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.936+0000 7f5306ffd640 1 --2- 172.21.15.156:0/1282109475 >> v2:172.21.15.203:3300/0 conn(0x7f53080b3be0 0x7f530806c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.937+0000 7f5307fff640 1 --2- 172.21.15.156:0/1282109475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f530806cb00 0x7f530815ddc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f53080b5430 tx=0x7f530002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d9300eea604ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:02.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.937+0000 7f53067fc640 1 -- 172.21.15.156:0/1282109475 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300039070 con 0x7f530806cb00 2024-03-29T04:10:02.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.937+0000 7f53067fc640 1 -- 172.21.15.156:0/1282109475 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530002cbf0 con 0x7f530806cb00 2024-03-29T04:10:02.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.937+0000 7f53067fc640 1 -- 172.21.15.156:0/1282109475 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300034440 con 0x7f530806cb00 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 -- 172.21.15.156:0/1282109475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f530806cb00 msgr2=0x7f530815ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 --2- 172.21.15.156:0/1282109475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f530806cb00 0x7f530815ddc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f53080b5430 tx=0x7f530002be40 comp rx=0 tx=0).stop 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 -- 172.21.15.156:0/1282109475 shutdown_connections 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 --2- 172.21.15.156:0/1282109475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f530806cb00 0x7f530815ddc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 --2- 172.21.15.156:0/1282109475 >> v2:172.21.15.203:3300/0 conn(0x7f53080b3be0 0x7f530806c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 -- 172.21.15.156:0/1282109475 >> 172.21.15.156:0/1282109475 conn(0x7f5308072180 msgr2=0x7f5308072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 -- 172.21.15.156:0/1282109475 shutdown_connections 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.938+0000 7f530de0d640 1 -- 172.21.15.156:0/1282109475 wait complete. 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.939+0000 7f530de0d640 1 Processor -- start 2024-03-29T04:10:02.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.939+0000 7f530de0d640 1 -- start start 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f530de0d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f530806cb00 0x7f5308143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f530de0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 0x7f53080ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f530de0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53081569c0 0x7f5308150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f530de0d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53080b4c00 con 0x7f53080b3be0 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f530de0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53080b4920 con 0x7f53081569c0 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f530de0d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f530815e300 con 0x7f530806cb00 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f5306ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 0x7f53080ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f5306ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 0x7f53080ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:53832/0 (socket says 172.21.15.156:53832) 2024-03-29T04:10:02.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f5306ffd640 1 -- 172.21.15.156:0/1457798921 learned_addr learned my addr 172.21.15.156:0/1457798921 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.940+0000 7f5307fff640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53081569c0 0x7f5308150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5306ffd640 1 -- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f530806cb00 msgr2=0x7f5308143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f53077fe640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f530806cb00 0x7f5308143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5306ffd640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f530806cb00 0x7f5308143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5306ffd640 1 -- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53081569c0 msgr2=0x7f5308150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5306ffd640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53081569c0 0x7f5308150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5306ffd640 1 -- 172.21.15.156:0/1457798921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530815a2b0 con 0x7f53080b3be0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5307fff640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53081569c0 0x7f5308150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f53077fe640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f530806cb00 0x7f5308143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f5306ffd640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 0x7f53080ccdd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e810 tx=0x7f52fc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00f040 con 0x7f53080b3be0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52fc004030 con 0x7f53080b3be0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530815a550 con 0x7f53080b3be0 2024-03-29T04:10:02.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc004400 con 0x7f53080b3be0 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.941+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530815ab10 con 0x7f53080b3be0 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.942+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5308070910 con 0x7f53080b3be0 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.942+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f52fc00c6d0 con 0x7f53080b3be0 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.943+0000 7f52effff640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f52f406c530 0x7f52f406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.943+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f52fc08b3a0 con 0x7f53080b3be0 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.943+0000 7f53077fe640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f52f406c530 0x7f52f406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:02.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.943+0000 7f53077fe640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f52f406c530 0x7f52f406e9f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000f60 tx=0x7f52f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:02.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:02.947+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5308070910 con 0x7f53080b3be0 2024-03-29T04:10:03.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:03.167+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "1", "yes_i_really_really_mean_it": true} v 0) -- 0x7f53080cf370 con 0x7f53080b3be0 2024-03-29T04:10:04.037 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.057+0000 7f52effff640 1 -- 172.21.15.156:0/1457798921 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "1", "yes_i_really_really_mean_it": true}]=0 inline data enabled v16) ==== 165+0+0 (secure 0 0 0) 0x7f52fc017020 con 0x7f53080b3be0 2024-03-29T04:10:04.037 INFO:teuthology.orchestra.run.smithi156.stderr:inline data enabled 2024-03-29T04:10:04.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.059+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f52f406c530 msgr2=0x7f52f406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:04.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.059+0000 7f530de0d640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f52f406c530 0x7f52f406e9f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000f60 tx=0x7f52f8009040 comp rx=0 tx=0).stop 2024-03-29T04:10:04.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.059+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 msgr2=0x7f53080ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:04.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.059+0000 7f530de0d640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 0x7f53080ccdd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e810 tx=0x7f52fc00ecd0 comp rx=0 tx=0).stop 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 shutdown_connections 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f52f406c530 0x7f52f406e9f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53081569c0 0x7f5308150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53080b3be0 0x7f53080ccdd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 --2- 172.21.15.156:0/1457798921 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f530806cb00 0x7f5308143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 >> 172.21.15.156:0/1457798921 conn(0x7f5308072180 msgr2=0x7f53080748a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 shutdown_connections 2024-03-29T04:10:04.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:04.060+0000 7f530de0d640 1 -- 172.21.15.156:0/1457798921 wait complete. 2024-03-29T04:10:04.054 DEBUG:tasks.cephfs.test_uninlining:fetching tarball 2024-03-29T04:10:04.054 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && wget http://download.ceph.com/qa/linux-5.4.tar.gz 2024-03-29T04:10:04.089 INFO:teuthology.orchestra.run.smithi156.stderr:--2024-03-29 04:10:04-- http://download.ceph.com/qa/linux-5.4.tar.gz 2024-03-29T04:10:04.098 INFO:teuthology.orchestra.run.smithi156.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-29T04:10:04.153 INFO:teuthology.orchestra.run.smithi156.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-29T04:10:04.189 INFO:teuthology.orchestra.run.smithi156.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-29T04:10:04.189 INFO:teuthology.orchestra.run.smithi156.stderr:Length: 172616875 (165M) [application/octet-stream] 2024-03-29T04:10:04.193 INFO:teuthology.orchestra.run.smithi156.stderr:Saving to: ‘linux-5.4.tar.gz’ 2024-03-29T04:10:04.195 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:10:04.261 INFO:teuthology.orchestra.run.smithi156.stderr: 0K .......... .......... .......... .......... .......... 0% 744K 3m46s 2024-03-29T04:10:04.332 INFO:teuthology.orchestra.run.smithi156.stderr: 50K .......... .......... .......... .......... .......... 0% 1.35M 2m54s 2024-03-29T04:10:04.333 INFO:teuthology.orchestra.run.smithi156.stderr: 100K .......... .......... .......... .......... .......... 0% 1.35M 2m37s 2024-03-29T04:10:04.371 INFO:teuthology.orchestra.run.smithi156.stderr: 150K .......... .......... .......... .......... .......... 0% 1.39M 2m27s 2024-03-29T04:10:04.371 INFO:teuthology.orchestra.run.smithi156.stderr: 200K .......... .......... .......... .......... .......... 0% 38.5M 1m58s 2024-03-29T04:10:04.371 INFO:teuthology.orchestra.run.smithi156.stderr: 250K .......... .......... .......... .......... .......... 0% 32.3M 99s 2024-03-29T04:10:04.404 INFO:teuthology.orchestra.run.smithi156.stderr: 300K .......... .......... .......... .......... .......... 0% 1.45M 1m41s 2024-03-29T04:10:04.406 INFO:teuthology.orchestra.run.smithi156.stderr: 350K .......... .......... .......... .......... .......... 0% 30.7M 89s 2024-03-29T04:10:04.407 INFO:teuthology.orchestra.run.smithi156.stderr: 400K .......... .......... .......... .......... .......... 0% 30.9M 80s 2024-03-29T04:10:04.409 INFO:teuthology.orchestra.run.smithi156.stderr: 450K .......... .......... .......... .......... .......... 0% 32.1M 72s 2024-03-29T04:10:04.410 INFO:teuthology.orchestra.run.smithi156.stderr: 500K .......... .......... .......... .......... .......... 0% 27.6M 66s 2024-03-29T04:10:04.438 INFO:teuthology.orchestra.run.smithi156.stderr: 550K .......... .......... .......... .......... .......... 0% 1.76M 69s 2024-03-29T04:10:04.440 INFO:teuthology.orchestra.run.smithi156.stderr: 600K .......... .......... .......... .......... .......... 0% 29.9M 64s 2024-03-29T04:10:04.441 INFO:teuthology.orchestra.run.smithi156.stderr: 650K .......... .......... .......... .......... .......... 0% 36.8M 60s 2024-03-29T04:10:04.443 INFO:teuthology.orchestra.run.smithi156.stderr: 700K .......... .......... .......... .......... .......... 0% 40.1M 56s 2024-03-29T04:10:04.443 INFO:teuthology.orchestra.run.smithi156.stderr: 750K .......... .......... .......... .......... .......... 0% 41.4M 53s 2024-03-29T04:10:04.445 INFO:teuthology.orchestra.run.smithi156.stderr: 800K .......... .......... .......... .......... .......... 0% 35.4M 50s 2024-03-29T04:10:04.446 INFO:teuthology.orchestra.run.smithi156.stderr: 850K .......... .......... .......... .......... .......... 0% 40.5M 47s 2024-03-29T04:10:04.448 INFO:teuthology.orchestra.run.smithi156.stderr: 900K .......... .......... .......... .......... .......... 0% 40.1M 45s 2024-03-29T04:10:04.474 INFO:teuthology.orchestra.run.smithi156.stderr: 950K .......... .......... .......... .......... .......... 0% 1.87M 47s 2024-03-29T04:10:04.475 INFO:teuthology.orchestra.run.smithi156.stderr: 1000K .......... .......... .......... .......... .......... 0% 27.7M 45s 2024-03-29T04:10:04.479 INFO:teuthology.orchestra.run.smithi156.stderr: 1050K .......... .......... .......... .......... .......... 0% 34.2M 43s 2024-03-29T04:10:04.480 INFO:teuthology.orchestra.run.smithi156.stderr: 1100K .......... .......... .......... .......... .......... 0% 37.5M 41s 2024-03-29T04:10:04.480 INFO:teuthology.orchestra.run.smithi156.stderr: 1150K .......... .......... .......... .......... .......... 0% 38.2M 40s 2024-03-29T04:10:04.480 INFO:teuthology.orchestra.run.smithi156.stderr: 1200K .......... .......... .......... .......... .......... 0% 33.6M 39s 2024-03-29T04:10:04.482 INFO:teuthology.orchestra.run.smithi156.stderr: 1250K .......... .......... .......... .......... .......... 0% 37.1M 37s 2024-03-29T04:10:04.483 INFO:teuthology.orchestra.run.smithi156.stderr: 1300K .......... .......... .......... .......... .......... 0% 38.3M 36s 2024-03-29T04:10:04.485 INFO:teuthology.orchestra.run.smithi156.stderr: 1350K .......... .......... .......... .......... .......... 0% 37.6M 35s 2024-03-29T04:10:04.486 INFO:teuthology.orchestra.run.smithi156.stderr: 1400K .......... .......... .......... .......... .......... 0% 31.7M 34s 2024-03-29T04:10:04.488 INFO:teuthology.orchestra.run.smithi156.stderr: 1450K .......... .......... .......... .......... .......... 0% 37.1M 33s 2024-03-29T04:10:04.490 INFO:teuthology.orchestra.run.smithi156.stderr: 1500K .......... .......... .......... .......... .......... 0% 34.7M 32s 2024-03-29T04:10:04.490 INFO:teuthology.orchestra.run.smithi156.stderr: 1550K .......... .......... .......... .......... .......... 0% 35.5M 31s 2024-03-29T04:10:04.492 INFO:teuthology.orchestra.run.smithi156.stderr: 1600K .......... .......... .......... .......... .......... 0% 31.4M 30s 2024-03-29T04:10:04.493 INFO:teuthology.orchestra.run.smithi156.stderr: 1650K .......... .......... .......... .......... .......... 1% 35.8M 29s 2024-03-29T04:10:04.494 INFO:teuthology.orchestra.run.smithi156.stderr: 1700K .......... .......... .......... .......... .......... 1% 36.0M 29s 2024-03-29T04:10:04.500 INFO:teuthology.orchestra.run.smithi156.stderr: 1750K .......... .......... .......... .......... .......... 1% 36.6M 28s 2024-03-29T04:10:04.500 INFO:teuthology.orchestra.run.smithi156.stderr: 1800K .......... .......... .......... .......... .......... 1% 32.2M 27s 2024-03-29T04:10:04.501 INFO:teuthology.orchestra.run.smithi156.stderr: 1850K .......... .......... .......... .......... .......... 1% 37.1M 27s 2024-03-29T04:10:04.509 INFO:teuthology.orchestra.run.smithi156.stderr: 1900K .......... .......... .......... .......... .......... 1% 4.72M 27s 2024-03-29T04:10:04.510 INFO:teuthology.orchestra.run.smithi156.stderr: 1950K .......... .......... .......... .......... .......... 1% 33.8M 26s 2024-03-29T04:10:04.513 INFO:teuthology.orchestra.run.smithi156.stderr: 2000K .......... .......... .......... .......... .......... 1% 32.9M 26s 2024-03-29T04:10:04.515 INFO:teuthology.orchestra.run.smithi156.stderr: 2050K .......... .......... .......... .......... .......... 1% 19.4M 25s 2024-03-29T04:10:04.516 INFO:teuthology.orchestra.run.smithi156.stderr: 2100K .......... .......... .......... .......... .......... 1% 30.9M 25s 2024-03-29T04:10:04.517 INFO:teuthology.orchestra.run.smithi156.stderr: 2150K .......... .......... .......... .......... .......... 1% 33.0M 25s 2024-03-29T04:10:04.519 INFO:teuthology.orchestra.run.smithi156.stderr: 2200K .......... .......... .......... .......... .......... 1% 33.4M 24s 2024-03-29T04:10:04.521 INFO:teuthology.orchestra.run.smithi156.stderr: 2250K .......... .......... .......... .......... .......... 1% 28.3M 24s 2024-03-29T04:10:04.522 INFO:teuthology.orchestra.run.smithi156.stderr: 2300K .......... .......... .......... .......... .......... 1% 34.0M 23s 2024-03-29T04:10:04.524 INFO:teuthology.orchestra.run.smithi156.stderr: 2350K .......... .......... .......... .......... .......... 1% 34.2M 23s 2024-03-29T04:10:04.525 INFO:teuthology.orchestra.run.smithi156.stderr: 2400K .......... .......... .......... .......... .......... 1% 33.8M 23s 2024-03-29T04:10:04.527 INFO:teuthology.orchestra.run.smithi156.stderr: 2450K .......... .......... .......... .......... .......... 1% 29.1M 22s 2024-03-29T04:10:04.528 INFO:teuthology.orchestra.run.smithi156.stderr: 2500K .......... .......... .......... .......... .......... 1% 31.8M 22s 2024-03-29T04:10:04.530 INFO:teuthology.orchestra.run.smithi156.stderr: 2550K .......... .......... .......... .......... .......... 1% 31.7M 21s 2024-03-29T04:10:04.537 INFO:teuthology.orchestra.run.smithi156.stderr: 2600K .......... .......... .......... .......... .......... 1% 6.43M 22s 2024-03-29T04:10:04.539 INFO:teuthology.orchestra.run.smithi156.stderr: 2650K .......... .......... .......... .......... .......... 1% 28.6M 21s 2024-03-29T04:10:04.541 INFO:teuthology.orchestra.run.smithi156.stderr: 2700K .......... .......... .......... .......... .......... 1% 30.7M 21s 2024-03-29T04:10:04.543 INFO:teuthology.orchestra.run.smithi156.stderr: 2750K .......... .......... .......... .......... .......... 1% 29.5M 21s 2024-03-29T04:10:04.544 INFO:teuthology.orchestra.run.smithi156.stderr: 2800K .......... .......... .......... .......... .......... 1% 26.7M 20s 2024-03-29T04:10:04.546 INFO:teuthology.orchestra.run.smithi156.stderr: 2850K .......... .......... .......... .......... .......... 1% 31.6M 20s 2024-03-29T04:10:04.548 INFO:teuthology.orchestra.run.smithi156.stderr: 2900K .......... .......... .......... .......... .......... 1% 30.3M 20s 2024-03-29T04:10:04.549 INFO:teuthology.orchestra.run.smithi156.stderr: 2950K .......... .......... .......... .......... .......... 1% 30.2M 20s 2024-03-29T04:10:04.551 INFO:teuthology.orchestra.run.smithi156.stderr: 3000K .......... .......... .......... .......... .......... 1% 26.7M 19s 2024-03-29T04:10:04.552 INFO:teuthology.orchestra.run.smithi156.stderr: 3050K .......... .......... .......... .......... .......... 1% 29.4M 19s 2024-03-29T04:10:04.554 INFO:teuthology.orchestra.run.smithi156.stderr: 3100K .......... .......... .......... .......... .......... 1% 30.5M 19s 2024-03-29T04:10:04.556 INFO:teuthology.orchestra.run.smithi156.stderr: 3150K .......... .......... .......... .......... .......... 1% 29.4M 19s 2024-03-29T04:10:04.557 INFO:teuthology.orchestra.run.smithi156.stderr: 3200K .......... .......... .......... .......... .......... 1% 26.7M 19s 2024-03-29T04:10:04.559 INFO:teuthology.orchestra.run.smithi156.stderr: 3250K .......... .......... .......... .......... .......... 1% 31.4M 18s 2024-03-29T04:10:04.560 INFO:teuthology.orchestra.run.smithi156.stderr: 3300K .......... .......... .......... .......... .......... 1% 31.0M 18s 2024-03-29T04:10:04.562 INFO:teuthology.orchestra.run.smithi156.stderr: 3350K .......... .......... .......... .......... .......... 2% 30.9M 18s 2024-03-29T04:10:04.564 INFO:teuthology.orchestra.run.smithi156.stderr: 3400K .......... .......... .......... .......... .......... 2% 24.9M 18s 2024-03-29T04:10:04.566 INFO:teuthology.orchestra.run.smithi156.stderr: 3450K .......... .......... .......... .......... .......... 2% 30.9M 18s 2024-03-29T04:10:04.567 INFO:teuthology.orchestra.run.smithi156.stderr: 3500K .......... .......... .......... .......... .......... 2% 30.5M 17s 2024-03-29T04:10:04.569 INFO:teuthology.orchestra.run.smithi156.stderr: 3550K .......... .......... .......... .......... .......... 2% 29.5M 17s 2024-03-29T04:10:04.571 INFO:teuthology.orchestra.run.smithi156.stderr: 3600K .......... .......... .......... .......... .......... 2% 27.0M 17s 2024-03-29T04:10:04.572 INFO:teuthology.orchestra.run.smithi156.stderr: 3650K .......... .......... .......... .......... .......... 2% 30.5M 17s 2024-03-29T04:10:04.574 INFO:teuthology.orchestra.run.smithi156.stderr: 3700K .......... .......... .......... .......... .......... 2% 27.3M 17s 2024-03-29T04:10:04.576 INFO:teuthology.orchestra.run.smithi156.stderr: 3750K .......... .......... .......... .......... .......... 2% 29.0M 17s 2024-03-29T04:10:04.578 INFO:teuthology.orchestra.run.smithi156.stderr: 3800K .......... .......... .......... .......... .......... 2% 25.1M 16s 2024-03-29T04:10:04.580 INFO:teuthology.orchestra.run.smithi156.stderr: 3850K .......... .......... .......... .......... .......... 2% 26.9M 16s 2024-03-29T04:10:04.581 INFO:teuthology.orchestra.run.smithi156.stderr: 3900K .......... .......... .......... .......... .......... 2% 26.2M 16s 2024-03-29T04:10:04.583 INFO:teuthology.orchestra.run.smithi156.stderr: 3950K .......... .......... .......... .......... .......... 2% 26.1M 16s 2024-03-29T04:10:04.585 INFO:teuthology.orchestra.run.smithi156.stderr: 4000K .......... .......... .......... .......... .......... 2% 23.7M 16s 2024-03-29T04:10:04.587 INFO:teuthology.orchestra.run.smithi156.stderr: 4050K .......... .......... .......... .......... .......... 2% 28.6M 16s 2024-03-29T04:10:04.588 INFO:teuthology.orchestra.run.smithi156.stderr: 4100K .......... .......... .......... .......... .......... 2% 36.9M 16s 2024-03-29T04:10:04.590 INFO:teuthology.orchestra.run.smithi156.stderr: 4150K .......... .......... .......... .......... .......... 2% 41.3M 16s 2024-03-29T04:10:04.591 INFO:teuthology.orchestra.run.smithi156.stderr: 4200K .......... .......... .......... .......... .......... 2% 35.5M 15s 2024-03-29T04:10:04.592 INFO:teuthology.orchestra.run.smithi156.stderr: 4250K .......... .......... .......... .......... .......... 2% 40.4M 15s 2024-03-29T04:10:04.593 INFO:teuthology.orchestra.run.smithi156.stderr: 4300K .......... .......... .......... .......... .......... 2% 41.6M 15s 2024-03-29T04:10:04.595 INFO:teuthology.orchestra.run.smithi156.stderr: 4350K .......... .......... .......... .......... .......... 2% 40.5M 15s 2024-03-29T04:10:04.596 INFO:teuthology.orchestra.run.smithi156.stderr: 4400K .......... .......... .......... .......... .......... 2% 34.1M 15s 2024-03-29T04:10:04.597 INFO:teuthology.orchestra.run.smithi156.stderr: 4450K .......... .......... .......... .......... .......... 2% 42.9M 15s 2024-03-29T04:10:04.598 INFO:teuthology.orchestra.run.smithi156.stderr: 4500K .......... .......... .......... .......... .......... 2% 42.4M 15s 2024-03-29T04:10:04.599 INFO:teuthology.orchestra.run.smithi156.stderr: 4550K .......... .......... .......... .......... .......... 2% 41.8M 14s 2024-03-29T04:10:04.601 INFO:teuthology.orchestra.run.smithi156.stderr: 4600K .......... .......... .......... .......... .......... 2% 35.8M 14s 2024-03-29T04:10:04.602 INFO:teuthology.orchestra.run.smithi156.stderr: 4650K .......... .......... .......... .......... .......... 2% 40.8M 14s 2024-03-29T04:10:04.603 INFO:teuthology.orchestra.run.smithi156.stderr: 4700K .......... .......... .......... .......... .......... 2% 37.9M 14s 2024-03-29T04:10:04.604 INFO:teuthology.orchestra.run.smithi156.stderr: 4750K .......... .......... .......... .......... .......... 2% 41.0M 14s 2024-03-29T04:10:04.612 INFO:teuthology.orchestra.run.smithi156.stderr: 4800K .......... .......... .......... .......... .......... 2% 6.33M 14s 2024-03-29T04:10:04.614 INFO:teuthology.orchestra.run.smithi156.stderr: 4850K .......... .......... .......... .......... .......... 2% 35.2M 14s 2024-03-29T04:10:04.615 INFO:teuthology.orchestra.run.smithi156.stderr: 4900K .......... .......... .......... .......... .......... 2% 29.4M 14s 2024-03-29T04:10:04.617 INFO:teuthology.orchestra.run.smithi156.stderr: 4950K .......... .......... .......... .......... .......... 2% 34.1M 14s 2024-03-29T04:10:04.618 INFO:teuthology.orchestra.run.smithi156.stderr: 5000K .......... .......... .......... .......... .......... 2% 32.4M 14s 2024-03-29T04:10:04.620 INFO:teuthology.orchestra.run.smithi156.stderr: 5050K .......... .......... .......... .......... .......... 3% 36.3M 14s 2024-03-29T04:10:04.621 INFO:teuthology.orchestra.run.smithi156.stderr: 5100K .......... .......... .......... .......... .......... 3% 26.4M 14s 2024-03-29T04:10:04.624 INFO:teuthology.orchestra.run.smithi156.stderr: 5150K .......... .......... .......... .......... .......... 3% 23.7M 14s 2024-03-29T04:10:04.625 INFO:teuthology.orchestra.run.smithi156.stderr: 5200K .......... .......... .......... .......... .......... 3% 24.8M 13s 2024-03-29T04:10:04.628 INFO:teuthology.orchestra.run.smithi156.stderr: 5250K .......... .......... .......... .......... .......... 3% 25.8M 13s 2024-03-29T04:10:04.629 INFO:teuthology.orchestra.run.smithi156.stderr: 5300K .......... .......... .......... .......... .......... 3% 22.5M 13s 2024-03-29T04:10:04.631 INFO:teuthology.orchestra.run.smithi156.stderr: 5350K .......... .......... .......... .......... .......... 3% 25.1M 13s 2024-03-29T04:10:04.633 INFO:teuthology.orchestra.run.smithi156.stderr: 5400K .......... .......... .......... .......... .......... 3% 26.8M 13s 2024-03-29T04:10:04.635 INFO:teuthology.orchestra.run.smithi156.stderr: 5450K .......... .......... .......... .......... .......... 3% 29.0M 13s 2024-03-29T04:10:04.637 INFO:teuthology.orchestra.run.smithi156.stderr: 5500K .......... .......... .......... .......... .......... 3% 25.5M 13s 2024-03-29T04:10:04.638 INFO:teuthology.orchestra.run.smithi156.stderr: 5550K .......... .......... .......... .......... .......... 3% 30.7M 13s 2024-03-29T04:10:04.640 INFO:teuthology.orchestra.run.smithi156.stderr: 5600K .......... .......... .......... .......... .......... 3% 35.0M 13s 2024-03-29T04:10:04.641 INFO:teuthology.orchestra.run.smithi156.stderr: 5650K .......... .......... .......... .......... .......... 3% 35.2M 13s 2024-03-29T04:10:04.643 INFO:teuthology.orchestra.run.smithi156.stderr: 5700K .......... .......... .......... .......... .......... 3% 29.5M 13s 2024-03-29T04:10:04.644 INFO:teuthology.orchestra.run.smithi156.stderr: 5750K .......... .......... .......... .......... .......... 3% 34.5M 13s 2024-03-29T04:10:04.646 INFO:teuthology.orchestra.run.smithi156.stderr: 5800K .......... .......... .......... .......... .......... 3% 35.1M 13s 2024-03-29T04:10:04.647 INFO:teuthology.orchestra.run.smithi156.stderr: 5850K .......... .......... .......... .......... .......... 3% 33.3M 13s 2024-03-29T04:10:04.649 INFO:teuthology.orchestra.run.smithi156.stderr: 5900K .......... .......... .......... .......... .......... 3% 29.8M 12s 2024-03-29T04:10:04.650 INFO:teuthology.orchestra.run.smithi156.stderr: 5950K .......... .......... .......... .......... .......... 3% 34.7M 12s 2024-03-29T04:10:04.652 INFO:teuthology.orchestra.run.smithi156.stderr: 6000K .......... .......... .......... .......... .......... 3% 33.3M 12s 2024-03-29T04:10:04.653 INFO:teuthology.orchestra.run.smithi156.stderr: 6050K .......... .......... .......... .......... .......... 3% 33.7M 12s 2024-03-29T04:10:04.655 INFO:teuthology.orchestra.run.smithi156.stderr: 6100K .......... .......... .......... .......... .......... 3% 29.7M 12s 2024-03-29T04:10:04.656 INFO:teuthology.orchestra.run.smithi156.stderr: 6150K .......... .......... .......... .......... .......... 3% 34.4M 12s 2024-03-29T04:10:04.658 INFO:teuthology.orchestra.run.smithi156.stderr: 6200K .......... .......... .......... .......... .......... 3% 34.0M 12s 2024-03-29T04:10:04.659 INFO:teuthology.orchestra.run.smithi156.stderr: 6250K .......... .......... .......... .......... .......... 3% 33.9M 12s 2024-03-29T04:10:04.661 INFO:teuthology.orchestra.run.smithi156.stderr: 6300K .......... .......... .......... .......... .......... 3% 28.6M 12s 2024-03-29T04:10:04.662 INFO:teuthology.orchestra.run.smithi156.stderr: 6350K .......... .......... .......... .......... .......... 3% 31.8M 12s 2024-03-29T04:10:04.664 INFO:teuthology.orchestra.run.smithi156.stderr: 6400K .......... .......... .......... .......... .......... 3% 31.9M 12s 2024-03-29T04:10:04.665 INFO:teuthology.orchestra.run.smithi156.stderr: 6450K .......... .......... .......... .......... .......... 3% 32.1M 12s 2024-03-29T04:10:04.667 INFO:teuthology.orchestra.run.smithi156.stderr: 6500K .......... .......... .......... .......... .......... 3% 28.2M 12s 2024-03-29T04:10:04.669 INFO:teuthology.orchestra.run.smithi156.stderr: 6550K .......... .......... .......... .......... .......... 3% 33.1M 12s 2024-03-29T04:10:04.670 INFO:teuthology.orchestra.run.smithi156.stderr: 6600K .......... .......... .......... .......... .......... 3% 31.6M 12s 2024-03-29T04:10:04.672 INFO:teuthology.orchestra.run.smithi156.stderr: 6650K .......... .......... .......... .......... .......... 3% 31.7M 12s 2024-03-29T04:10:04.673 INFO:teuthology.orchestra.run.smithi156.stderr: 6700K .......... .......... .......... .......... .......... 4% 28.0M 12s 2024-03-29T04:10:04.675 INFO:teuthology.orchestra.run.smithi156.stderr: 6750K .......... .......... .......... .......... .......... 4% 32.4M 11s 2024-03-29T04:10:04.676 INFO:teuthology.orchestra.run.smithi156.stderr: 6800K .......... .......... .......... .......... .......... 4% 31.6M 11s 2024-03-29T04:10:04.678 INFO:teuthology.orchestra.run.smithi156.stderr: 6850K .......... .......... .......... .......... .......... 4% 31.3M 11s 2024-03-29T04:10:04.683 INFO:teuthology.orchestra.run.smithi156.stderr: 6900K .......... .......... .......... .......... .......... 4% 27.5M 11s 2024-03-29T04:10:04.683 INFO:teuthology.orchestra.run.smithi156.stderr: 6950K .......... .......... .......... .......... .......... 4% 31.0M 11s 2024-03-29T04:10:04.683 INFO:teuthology.orchestra.run.smithi156.stderr: 7000K .......... .......... .......... .......... .......... 4% 30.5M 11s 2024-03-29T04:10:04.685 INFO:teuthology.orchestra.run.smithi156.stderr: 7050K .......... .......... .......... .......... .......... 4% 31.1M 11s 2024-03-29T04:10:04.686 INFO:teuthology.orchestra.run.smithi156.stderr: 7100K .......... .......... .......... .......... .......... 4% 26.4M 11s 2024-03-29T04:10:04.688 INFO:teuthology.orchestra.run.smithi156.stderr: 7150K .......... .......... .......... .......... .......... 4% 31.0M 11s 2024-03-29T04:10:04.689 INFO:teuthology.orchestra.run.smithi156.stderr: 7200K .......... .......... .......... .......... .......... 4% 30.8M 11s 2024-03-29T04:10:04.691 INFO:teuthology.orchestra.run.smithi156.stderr: 7250K .......... .......... .......... .......... .......... 4% 30.7M 11s 2024-03-29T04:10:04.693 INFO:teuthology.orchestra.run.smithi156.stderr: 7300K .......... .......... .......... .......... .......... 4% 25.9M 11s 2024-03-29T04:10:04.695 INFO:teuthology.orchestra.run.smithi156.stderr: 7350K .......... .......... .......... .......... .......... 4% 30.2M 11s 2024-03-29T04:10:04.696 INFO:teuthology.orchestra.run.smithi156.stderr: 7400K .......... .......... .......... .......... .......... 4% 30.1M 11s 2024-03-29T04:10:04.698 INFO:teuthology.orchestra.run.smithi156.stderr: 7450K .......... .......... .......... .......... .......... 4% 29.6M 11s 2024-03-29T04:10:04.700 INFO:teuthology.orchestra.run.smithi156.stderr: 7500K .......... .......... .......... .......... .......... 4% 25.6M 11s 2024-03-29T04:10:04.701 INFO:teuthology.orchestra.run.smithi156.stderr: 7550K .......... .......... .......... .......... .......... 4% 30.5M 11s 2024-03-29T04:10:04.703 INFO:teuthology.orchestra.run.smithi156.stderr: 7600K .......... .......... .......... .......... .......... 4% 29.4M 11s 2024-03-29T04:10:04.705 INFO:teuthology.orchestra.run.smithi156.stderr: 7650K .......... .......... .......... .......... .......... 4% 30.3M 11s 2024-03-29T04:10:04.706 INFO:teuthology.orchestra.run.smithi156.stderr: 7700K .......... .......... .......... .......... .......... 4% 25.3M 11s 2024-03-29T04:10:04.708 INFO:teuthology.orchestra.run.smithi156.stderr: 7750K .......... .......... .......... .......... .......... 4% 29.9M 11s 2024-03-29T04:10:04.710 INFO:teuthology.orchestra.run.smithi156.stderr: 7800K .......... .......... .......... .......... .......... 4% 29.8M 11s 2024-03-29T04:10:04.712 INFO:teuthology.orchestra.run.smithi156.stderr: 7850K .......... .......... .......... .......... .......... 4% 28.6M 11s 2024-03-29T04:10:04.714 INFO:teuthology.orchestra.run.smithi156.stderr: 7900K .......... .......... .......... .......... .......... 4% 24.8M 11s 2024-03-29T04:10:04.715 INFO:teuthology.orchestra.run.smithi156.stderr: 7950K .......... .......... .......... .......... .......... 4% 28.6M 10s 2024-03-29T04:10:04.717 INFO:teuthology.orchestra.run.smithi156.stderr: 8000K .......... .......... .......... .......... .......... 4% 28.6M 10s 2024-03-29T04:10:04.719 INFO:teuthology.orchestra.run.smithi156.stderr: 8050K .......... .......... .......... .......... .......... 4% 29.2M 10s 2024-03-29T04:10:04.721 INFO:teuthology.orchestra.run.smithi156.stderr: 8100K .......... .......... .......... .......... .......... 4% 23.9M 10s 2024-03-29T04:10:04.723 INFO:teuthology.orchestra.run.smithi156.stderr: 8150K .......... .......... .......... .......... .......... 4% 27.8M 10s 2024-03-29T04:10:04.725 INFO:teuthology.orchestra.run.smithi156.stderr: 8200K .......... .......... .......... .......... .......... 4% 29.7M 10s 2024-03-29T04:10:04.726 INFO:teuthology.orchestra.run.smithi156.stderr: 8250K .......... .......... .......... .......... .......... 4% 31.2M 10s 2024-03-29T04:10:04.728 INFO:teuthology.orchestra.run.smithi156.stderr: 8300K .......... .......... .......... .......... .......... 4% 20.2M 10s 2024-03-29T04:10:04.729 INFO:teuthology.orchestra.run.smithi156.stderr: 8350K .......... .......... .......... .......... .......... 4% 36.6M 10s 2024-03-29T04:10:04.731 INFO:teuthology.orchestra.run.smithi156.stderr: 8400K .......... .......... .......... .......... .......... 5% 36.4M 10s 2024-03-29T04:10:04.732 INFO:teuthology.orchestra.run.smithi156.stderr: 8450K .......... .......... .......... .......... .......... 5% 33.9M 10s 2024-03-29T04:10:04.734 INFO:teuthology.orchestra.run.smithi156.stderr: 8500K .......... .......... .......... .......... .......... 5% 25.5M 10s 2024-03-29T04:10:04.736 INFO:teuthology.orchestra.run.smithi156.stderr: 8550K .......... .......... .......... .......... .......... 5% 24.8M 10s 2024-03-29T04:10:04.738 INFO:teuthology.orchestra.run.smithi156.stderr: 8600K .......... .......... .......... .......... .......... 5% 26.1M 10s 2024-03-29T04:10:04.740 INFO:teuthology.orchestra.run.smithi156.stderr: 8650K .......... .......... .......... .......... .......... 5% 27.4M 10s 2024-03-29T04:10:04.742 INFO:teuthology.orchestra.run.smithi156.stderr: 8700K .......... .......... .......... .......... .......... 5% 22.1M 10s 2024-03-29T04:10:04.744 INFO:teuthology.orchestra.run.smithi156.stderr: 8750K .......... .......... .......... .......... .......... 5% 27.3M 10s 2024-03-29T04:10:04.745 INFO:teuthology.orchestra.run.smithi156.stderr: 8800K .......... .......... .......... .......... .......... 5% 31.9M 10s 2024-03-29T04:10:04.750 INFO:teuthology.orchestra.run.smithi156.stderr: 8850K .......... .......... .......... .......... .......... 5% 32.7M 10s 2024-03-29T04:10:04.750 INFO:teuthology.orchestra.run.smithi156.stderr: 8900K .......... .......... .......... .......... .......... 5% 28.5M 10s 2024-03-29T04:10:04.750 INFO:teuthology.orchestra.run.smithi156.stderr: 8950K .......... .......... .......... .......... .......... 5% 32.6M 10s 2024-03-29T04:10:04.752 INFO:teuthology.orchestra.run.smithi156.stderr: 9000K .......... .......... .......... .......... .......... 5% 33.7M 10s 2024-03-29T04:10:04.753 INFO:teuthology.orchestra.run.smithi156.stderr: 9050K .......... .......... .......... .......... .......... 5% 32.5M 10s 2024-03-29T04:10:04.755 INFO:teuthology.orchestra.run.smithi156.stderr: 9100K .......... .......... .......... .......... .......... 5% 28.1M 10s 2024-03-29T04:10:04.756 INFO:teuthology.orchestra.run.smithi156.stderr: 9150K .......... .......... .......... .......... .......... 5% 32.5M 10s 2024-03-29T04:10:04.759 INFO:teuthology.orchestra.run.smithi156.stderr: 9200K .......... .......... .......... .......... .......... 5% 33.0M 10s 2024-03-29T04:10:04.760 INFO:teuthology.orchestra.run.smithi156.stderr: 9250K .......... .......... .......... .......... .......... 5% 32.0M 10s 2024-03-29T04:10:04.761 INFO:teuthology.orchestra.run.smithi156.stderr: 9300K .......... .......... .......... .......... .......... 5% 29.3M 10s 2024-03-29T04:10:04.763 INFO:teuthology.orchestra.run.smithi156.stderr: 9350K .......... .......... .......... .......... .......... 5% 32.7M 10s 2024-03-29T04:10:04.764 INFO:teuthology.orchestra.run.smithi156.stderr: 9400K .......... .......... .......... .......... .......... 5% 32.8M 10s 2024-03-29T04:10:04.766 INFO:teuthology.orchestra.run.smithi156.stderr: 9450K .......... .......... .......... .......... .......... 5% 32.6M 10s 2024-03-29T04:10:04.767 INFO:teuthology.orchestra.run.smithi156.stderr: 9500K .......... .......... .......... .......... .......... 5% 28.0M 10s 2024-03-29T04:10:04.769 INFO:teuthology.orchestra.run.smithi156.stderr: 9550K .......... .......... .......... .......... .......... 5% 29.0M 10s 2024-03-29T04:10:04.770 INFO:teuthology.orchestra.run.smithi156.stderr: 9600K .......... .......... .......... .......... .......... 5% 29.4M 10s 2024-03-29T04:10:04.773 INFO:teuthology.orchestra.run.smithi156.stderr: 9650K .......... .......... .......... .......... .......... 5% 29.4M 9s 2024-03-29T04:10:04.774 INFO:teuthology.orchestra.run.smithi156.stderr: 9700K .......... .......... .......... .......... .......... 5% 25.9M 9s 2024-03-29T04:10:04.775 INFO:teuthology.orchestra.run.smithi156.stderr: 9750K .......... .......... .......... .......... .......... 5% 29.6M 9s 2024-03-29T04:10:04.777 INFO:teuthology.orchestra.run.smithi156.stderr: 9800K .......... .......... .......... .......... .......... 5% 29.5M 9s 2024-03-29T04:10:04.779 INFO:teuthology.orchestra.run.smithi156.stderr: 9850K .......... .......... .......... .......... .......... 5% 30.0M 9s 2024-03-29T04:10:04.781 INFO:teuthology.orchestra.run.smithi156.stderr: 9900K .......... .......... .......... .......... .......... 5% 27.2M 9s 2024-03-29T04:10:04.782 INFO:teuthology.orchestra.run.smithi156.stderr: 9950K .......... .......... .......... .......... .......... 5% 31.1M 9s 2024-03-29T04:10:04.784 INFO:teuthology.orchestra.run.smithi156.stderr: 10000K .......... .......... .......... .......... .......... 5% 30.8M 9s 2024-03-29T04:10:04.785 INFO:teuthology.orchestra.run.smithi156.stderr: 10050K .......... .......... .......... .......... .......... 5% 31.2M 9s 2024-03-29T04:10:04.787 INFO:teuthology.orchestra.run.smithi156.stderr: 10100K .......... .......... .......... .......... .......... 6% 27.3M 9s 2024-03-29T04:10:04.789 INFO:teuthology.orchestra.run.smithi156.stderr: 10150K .......... .......... .......... .......... .......... 6% 29.3M 9s 2024-03-29T04:10:04.791 INFO:teuthology.orchestra.run.smithi156.stderr: 10200K .......... .......... .......... .......... .......... 6% 28.4M 9s 2024-03-29T04:10:04.792 INFO:teuthology.orchestra.run.smithi156.stderr: 10250K .......... .......... .......... .......... .......... 6% 28.4M 9s 2024-03-29T04:10:04.794 INFO:teuthology.orchestra.run.smithi156.stderr: 10300K .......... .......... .......... .......... .......... 6% 24.5M 9s 2024-03-29T04:10:04.796 INFO:teuthology.orchestra.run.smithi156.stderr: 10350K .......... .......... .......... .......... .......... 6% 28.5M 9s 2024-03-29T04:10:04.798 INFO:teuthology.orchestra.run.smithi156.stderr: 10400K .......... .......... .......... .......... .......... 6% 28.6M 9s 2024-03-29T04:10:04.800 INFO:teuthology.orchestra.run.smithi156.stderr: 10450K .......... .......... .......... .......... .......... 6% 27.9M 9s 2024-03-29T04:10:04.801 INFO:teuthology.orchestra.run.smithi156.stderr: 10500K .......... .......... .......... .......... .......... 6% 24.1M 9s 2024-03-29T04:10:04.803 INFO:teuthology.orchestra.run.smithi156.stderr: 10550K .......... .......... .......... .......... .......... 6% 31.8M 9s 2024-03-29T04:10:04.804 INFO:teuthology.orchestra.run.smithi156.stderr: 10600K .......... .......... .......... .......... .......... 6% 34.3M 9s 2024-03-29T04:10:04.806 INFO:teuthology.orchestra.run.smithi156.stderr: 10650K .......... .......... .......... .......... .......... 6% 32.0M 9s 2024-03-29T04:10:04.808 INFO:teuthology.orchestra.run.smithi156.stderr: 10700K .......... .......... .......... .......... .......... 6% 29.6M 9s 2024-03-29T04:10:04.809 INFO:teuthology.orchestra.run.smithi156.stderr: 10750K .......... .......... .......... .......... .......... 6% 32.5M 9s 2024-03-29T04:10:04.810 INFO:teuthology.orchestra.run.smithi156.stderr: 10800K .......... .......... .......... .......... .......... 6% 34.1M 9s 2024-03-29T04:10:04.812 INFO:teuthology.orchestra.run.smithi156.stderr: 10850K .......... .......... .......... .......... .......... 6% 31.5M 9s 2024-03-29T04:10:04.814 INFO:teuthology.orchestra.run.smithi156.stderr: 10900K .......... .......... .......... .......... .......... 6% 27.5M 9s 2024-03-29T04:10:04.815 INFO:teuthology.orchestra.run.smithi156.stderr: 10950K .......... .......... .......... .......... .......... 6% 28.8M 9s 2024-03-29T04:10:04.817 INFO:teuthology.orchestra.run.smithi156.stderr: 11000K .......... .......... .......... .......... .......... 6% 29.2M 9s 2024-03-29T04:10:04.819 INFO:teuthology.orchestra.run.smithi156.stderr: 11050K .......... .......... .......... .......... .......... 6% 29.1M 9s 2024-03-29T04:10:04.821 INFO:teuthology.orchestra.run.smithi156.stderr: 11100K .......... .......... .......... .......... .......... 6% 25.1M 9s 2024-03-29T04:10:04.822 INFO:teuthology.orchestra.run.smithi156.stderr: 11150K .......... .......... .......... .......... .......... 6% 28.9M 9s 2024-03-29T04:10:04.824 INFO:teuthology.orchestra.run.smithi156.stderr: 11200K .......... .......... .......... .......... .......... 6% 29.0M 9s 2024-03-29T04:10:04.826 INFO:teuthology.orchestra.run.smithi156.stderr: 11250K .......... .......... .......... .......... .......... 6% 28.7M 9s 2024-03-29T04:10:04.828 INFO:teuthology.orchestra.run.smithi156.stderr: 11300K .......... .......... .......... .......... .......... 6% 23.8M 9s 2024-03-29T04:10:04.830 INFO:teuthology.orchestra.run.smithi156.stderr: 11350K .......... .......... .......... .......... .......... 6% 28.4M 9s 2024-03-29T04:10:04.831 INFO:teuthology.orchestra.run.smithi156.stderr: 11400K .......... .......... .......... .......... .......... 6% 28.2M 9s 2024-03-29T04:10:04.834 INFO:teuthology.orchestra.run.smithi156.stderr: 11450K .......... .......... .......... .......... .......... 6% 27.5M 9s 2024-03-29T04:10:04.835 INFO:teuthology.orchestra.run.smithi156.stderr: 11500K .......... .......... .......... .......... .......... 6% 25.1M 9s 2024-03-29T04:10:04.837 INFO:teuthology.orchestra.run.smithi156.stderr: 11550K .......... .......... .......... .......... .......... 6% 28.0M 9s 2024-03-29T04:10:04.839 INFO:teuthology.orchestra.run.smithi156.stderr: 11600K .......... .......... .......... .......... .......... 6% 29.4M 9s 2024-03-29T04:10:04.841 INFO:teuthology.orchestra.run.smithi156.stderr: 11650K .......... .......... .......... .......... .......... 6% 26.4M 9s 2024-03-29T04:10:04.842 INFO:teuthology.orchestra.run.smithi156.stderr: 11700K .......... .......... .......... .......... .......... 6% 23.5M 9s 2024-03-29T04:10:04.845 INFO:teuthology.orchestra.run.smithi156.stderr: 11750K .......... .......... .......... .......... .......... 7% 27.2M 9s 2024-03-29T04:10:04.846 INFO:teuthology.orchestra.run.smithi156.stderr: 11800K .......... .......... .......... .......... .......... 7% 27.1M 9s 2024-03-29T04:10:04.848 INFO:teuthology.orchestra.run.smithi156.stderr: 11850K .......... .......... .......... .......... .......... 7% 27.0M 9s 2024-03-29T04:10:04.850 INFO:teuthology.orchestra.run.smithi156.stderr: 11900K .......... .......... .......... .......... .......... 7% 23.6M 9s 2024-03-29T04:10:04.852 INFO:teuthology.orchestra.run.smithi156.stderr: 11950K .......... .......... .......... .......... .......... 7% 26.7M 9s 2024-03-29T04:10:04.854 INFO:teuthology.orchestra.run.smithi156.stderr: 12000K .......... .......... .......... .......... .......... 7% 25.3M 9s 2024-03-29T04:10:04.856 INFO:teuthology.orchestra.run.smithi156.stderr: 12050K .......... .......... .......... .......... .......... 7% 25.6M 9s 2024-03-29T04:10:04.858 INFO:teuthology.orchestra.run.smithi156.stderr: 12100K .......... .......... .......... .......... .......... 7% 20.6M 9s 2024-03-29T04:10:04.860 INFO:teuthology.orchestra.run.smithi156.stderr: 12150K .......... .......... .......... .......... .......... 7% 25.1M 9s 2024-03-29T04:10:04.862 INFO:teuthology.orchestra.run.smithi156.stderr: 12200K .......... .......... .......... .......... .......... 7% 25.0M 9s 2024-03-29T04:10:04.864 INFO:teuthology.orchestra.run.smithi156.stderr: 12250K .......... .......... .......... .......... .......... 7% 25.3M 9s 2024-03-29T04:10:04.866 INFO:teuthology.orchestra.run.smithi156.stderr: 12300K .......... .......... .......... .......... .......... 7% 20.6M 9s 2024-03-29T04:10:04.868 INFO:teuthology.orchestra.run.smithi156.stderr: 12350K .......... .......... .......... .......... .......... 7% 20.9M 9s 2024-03-29T04:10:04.870 INFO:teuthology.orchestra.run.smithi156.stderr: 12400K .......... .......... .......... .......... .......... 7% 31.5M 8s 2024-03-29T04:10:04.871 INFO:teuthology.orchestra.run.smithi156.stderr: 12450K .......... .......... .......... .......... .......... 7% 39.7M 8s 2024-03-29T04:10:04.873 INFO:teuthology.orchestra.run.smithi156.stderr: 12500K .......... .......... .......... .......... .......... 7% 34.6M 8s 2024-03-29T04:10:04.874 INFO:teuthology.orchestra.run.smithi156.stderr: 12550K .......... .......... .......... .......... .......... 7% 32.1M 8s 2024-03-29T04:10:04.876 INFO:teuthology.orchestra.run.smithi156.stderr: 12600K .......... .......... .......... .......... .......... 7% 32.9M 8s 2024-03-29T04:10:04.877 INFO:teuthology.orchestra.run.smithi156.stderr: 12650K .......... .......... .......... .......... .......... 7% 34.0M 8s 2024-03-29T04:10:04.879 INFO:teuthology.orchestra.run.smithi156.stderr: 12700K .......... .......... .......... .......... .......... 7% 24.6M 8s 2024-03-29T04:10:04.881 INFO:teuthology.orchestra.run.smithi156.stderr: 12750K .......... .......... .......... .......... .......... 7% 30.6M 8s 2024-03-29T04:10:04.888 INFO:teuthology.orchestra.run.smithi156.stderr: 12800K .......... .......... .......... .......... .......... 7% 24.8M 8s 2024-03-29T04:10:04.888 INFO:teuthology.orchestra.run.smithi156.stderr: 12850K .......... .......... .......... .......... .......... 7% 30.9M 8s 2024-03-29T04:10:04.888 INFO:teuthology.orchestra.run.smithi156.stderr: 12900K .......... .......... .......... .......... .......... 7% 27.9M 8s 2024-03-29T04:10:04.888 INFO:teuthology.orchestra.run.smithi156.stderr: 12950K .......... .......... .......... .......... .......... 7% 32.7M 8s 2024-03-29T04:10:04.889 INFO:teuthology.orchestra.run.smithi156.stderr: 13000K .......... .......... .......... .......... .......... 7% 31.3M 8s 2024-03-29T04:10:04.891 INFO:teuthology.orchestra.run.smithi156.stderr: 13050K .......... .......... .......... .......... .......... 7% 31.8M 8s 2024-03-29T04:10:04.892 INFO:teuthology.orchestra.run.smithi156.stderr: 13100K .......... .......... .......... .......... .......... 7% 27.5M 8s 2024-03-29T04:10:04.894 INFO:teuthology.orchestra.run.smithi156.stderr: 13150K .......... .......... .......... .......... .......... 7% 31.9M 8s 2024-03-29T04:10:04.895 INFO:teuthology.orchestra.run.smithi156.stderr: 13200K .......... .......... .......... .......... .......... 7% 31.2M 8s 2024-03-29T04:10:04.897 INFO:teuthology.orchestra.run.smithi156.stderr: 13250K .......... .......... .......... .......... .......... 7% 31.9M 8s 2024-03-29T04:10:04.899 INFO:teuthology.orchestra.run.smithi156.stderr: 13300K .......... .......... .......... .......... .......... 7% 27.7M 8s 2024-03-29T04:10:04.900 INFO:teuthology.orchestra.run.smithi156.stderr: 13350K .......... .......... .......... .......... .......... 7% 31.1M 8s 2024-03-29T04:10:04.902 INFO:teuthology.orchestra.run.smithi156.stderr: 13400K .......... .......... .......... .......... .......... 7% 31.4M 8s 2024-03-29T04:10:04.904 INFO:teuthology.orchestra.run.smithi156.stderr: 13450K .......... .......... .......... .......... .......... 8% 30.8M 8s 2024-03-29T04:10:04.905 INFO:teuthology.orchestra.run.smithi156.stderr: 13500K .......... .......... .......... .......... .......... 8% 26.8M 8s 2024-03-29T04:10:04.907 INFO:teuthology.orchestra.run.smithi156.stderr: 13550K .......... .......... .......... .......... .......... 8% 31.5M 8s 2024-03-29T04:10:04.909 INFO:teuthology.orchestra.run.smithi156.stderr: 13600K .......... .......... .......... .......... .......... 8% 29.9M 8s 2024-03-29T04:10:04.910 INFO:teuthology.orchestra.run.smithi156.stderr: 13650K .......... .......... .......... .......... .......... 8% 27.8M 8s 2024-03-29T04:10:04.912 INFO:teuthology.orchestra.run.smithi156.stderr: 13700K .......... .......... .......... .......... .......... 8% 23.9M 8s 2024-03-29T04:10:04.914 INFO:teuthology.orchestra.run.smithi156.stderr: 13750K .......... .......... .......... .......... .......... 8% 28.3M 8s 2024-03-29T04:10:04.916 INFO:teuthology.orchestra.run.smithi156.stderr: 13800K .......... .......... .......... .......... .......... 8% 27.4M 8s 2024-03-29T04:10:04.918 INFO:teuthology.orchestra.run.smithi156.stderr: 13850K .......... .......... .......... .......... .......... 8% 28.4M 8s 2024-03-29T04:10:04.920 INFO:teuthology.orchestra.run.smithi156.stderr: 13900K .......... .......... .......... .......... .......... 8% 23.2M 8s 2024-03-29T04:10:04.921 INFO:teuthology.orchestra.run.smithi156.stderr: 13950K .......... .......... .......... .......... .......... 8% 26.2M 8s 2024-03-29T04:10:04.923 INFO:teuthology.orchestra.run.smithi156.stderr: 14000K .......... .......... .......... .......... .......... 8% 26.9M 8s 2024-03-29T04:10:04.925 INFO:teuthology.orchestra.run.smithi156.stderr: 14050K .......... .......... .......... .......... .......... 8% 27.4M 8s 2024-03-29T04:10:04.927 INFO:teuthology.orchestra.run.smithi156.stderr: 14100K .......... .......... .......... .......... .......... 8% 22.6M 8s 2024-03-29T04:10:04.929 INFO:teuthology.orchestra.run.smithi156.stderr: 14150K .......... .......... .......... .......... .......... 8% 27.4M 8s 2024-03-29T04:10:04.931 INFO:teuthology.orchestra.run.smithi156.stderr: 14200K .......... .......... .......... .......... .......... 8% 26.6M 8s 2024-03-29T04:10:04.933 INFO:teuthology.orchestra.run.smithi156.stderr: 14250K .......... .......... .......... .......... .......... 8% 26.4M 8s 2024-03-29T04:10:04.935 INFO:teuthology.orchestra.run.smithi156.stderr: 14300K .......... .......... .......... .......... .......... 8% 21.8M 8s 2024-03-29T04:10:04.937 INFO:teuthology.orchestra.run.smithi156.stderr: 14350K .......... .......... .......... .......... .......... 8% 25.3M 8s 2024-03-29T04:10:04.939 INFO:teuthology.orchestra.run.smithi156.stderr: 14400K .......... .......... .......... .......... .......... 8% 25.4M 8s 2024-03-29T04:10:04.941 INFO:teuthology.orchestra.run.smithi156.stderr: 14450K .......... .......... .......... .......... .......... 8% 27.7M 8s 2024-03-29T04:10:04.943 INFO:teuthology.orchestra.run.smithi156.stderr: 14500K .......... .......... .......... .......... .......... 8% 24.1M 8s 2024-03-29T04:10:04.944 INFO:teuthology.orchestra.run.smithi156.stderr: 14550K .......... .......... .......... .......... .......... 8% 27.8M 8s 2024-03-29T04:10:04.947 INFO:teuthology.orchestra.run.smithi156.stderr: 14600K .......... .......... .......... .......... .......... 8% 22.5M 8s 2024-03-29T04:10:04.948 INFO:teuthology.orchestra.run.smithi156.stderr: 14650K .......... .......... .......... .......... .......... 8% 28.7M 8s 2024-03-29T04:10:04.950 INFO:teuthology.orchestra.run.smithi156.stderr: 14700K .......... .......... .......... .......... .......... 8% 26.5M 8s 2024-03-29T04:10:04.952 INFO:teuthology.orchestra.run.smithi156.stderr: 14750K .......... .......... .......... .......... .......... 8% 31.5M 8s 2024-03-29T04:10:04.953 INFO:teuthology.orchestra.run.smithi156.stderr: 14800K .......... .......... .......... .......... .......... 8% 31.4M 8s 2024-03-29T04:10:04.955 INFO:teuthology.orchestra.run.smithi156.stderr: 14850K .......... .......... .......... .......... .......... 8% 31.9M 8s 2024-03-29T04:10:04.956 INFO:teuthology.orchestra.run.smithi156.stderr: 14900K .......... .......... .......... .......... .......... 8% 26.7M 8s 2024-03-29T04:10:04.958 INFO:teuthology.orchestra.run.smithi156.stderr: 14950K .......... .......... .......... .......... .......... 8% 31.0M 8s 2024-03-29T04:10:04.960 INFO:teuthology.orchestra.run.smithi156.stderr: 15000K .......... .......... .......... .......... .......... 8% 30.4M 8s 2024-03-29T04:10:04.961 INFO:teuthology.orchestra.run.smithi156.stderr: 15050K .......... .......... .......... .......... .......... 8% 30.2M 8s 2024-03-29T04:10:04.963 INFO:teuthology.orchestra.run.smithi156.stderr: 15100K .......... .......... .......... .......... .......... 8% 26.6M 8s 2024-03-29T04:10:04.965 INFO:teuthology.orchestra.run.smithi156.stderr: 15150K .......... .......... .......... .......... .......... 9% 30.9M 8s 2024-03-29T04:10:04.966 INFO:teuthology.orchestra.run.smithi156.stderr: 15200K .......... .......... .......... .......... .......... 9% 30.9M 8s 2024-03-29T04:10:04.968 INFO:teuthology.orchestra.run.smithi156.stderr: 15250K .......... .......... .......... .......... .......... 9% 30.5M 8s 2024-03-29T04:10:04.970 INFO:teuthology.orchestra.run.smithi156.stderr: 15300K .......... .......... .......... .......... .......... 9% 25.7M 8s 2024-03-29T04:10:04.972 INFO:teuthology.orchestra.run.smithi156.stderr: 15350K .......... .......... .......... .......... .......... 9% 26.4M 8s 2024-03-29T04:10:04.973 INFO:teuthology.orchestra.run.smithi156.stderr: 15400K .......... .......... .......... .......... .......... 9% 25.3M 8s 2024-03-29T04:10:04.976 INFO:teuthology.orchestra.run.smithi156.stderr: 15450K .......... .......... .......... .......... .......... 9% 26.1M 8s 2024-03-29T04:10:04.978 INFO:teuthology.orchestra.run.smithi156.stderr: 15500K .......... .......... .......... .......... .......... 9% 21.3M 8s 2024-03-29T04:10:04.980 INFO:teuthology.orchestra.run.smithi156.stderr: 15550K .......... .......... .......... .......... .......... 9% 25.1M 8s 2024-03-29T04:10:04.982 INFO:teuthology.orchestra.run.smithi156.stderr: 15600K .......... .......... .......... .......... .......... 9% 25.6M 8s 2024-03-29T04:10:04.984 INFO:teuthology.orchestra.run.smithi156.stderr: 15650K .......... .......... .......... .......... .......... 9% 24.1M 8s 2024-03-29T04:10:04.986 INFO:teuthology.orchestra.run.smithi156.stderr: 15700K .......... .......... .......... .......... .......... 9% 19.3M 8s 2024-03-29T04:10:04.988 INFO:teuthology.orchestra.run.smithi156.stderr: 15750K .......... .......... .......... .......... .......... 9% 22.9M 8s 2024-03-29T04:10:04.990 INFO:teuthology.orchestra.run.smithi156.stderr: 15800K .......... .......... .......... .......... .......... 9% 26.7M 8s 2024-03-29T04:10:04.992 INFO:teuthology.orchestra.run.smithi156.stderr: 15850K .......... .......... .......... .......... .......... 9% 26.1M 8s 2024-03-29T04:10:04.994 INFO:teuthology.orchestra.run.smithi156.stderr: 15900K .......... .......... .......... .......... .......... 9% 22.9M 8s 2024-03-29T04:10:04.996 INFO:teuthology.orchestra.run.smithi156.stderr: 15950K .......... .......... .......... .......... .......... 9% 26.2M 8s 2024-03-29T04:10:04.998 INFO:teuthology.orchestra.run.smithi156.stderr: 16000K .......... .......... .......... .......... .......... 9% 24.7M 8s 2024-03-29T04:10:05.000 INFO:teuthology.orchestra.run.smithi156.stderr: 16050K .......... .......... .......... .......... .......... 9% 23.0M 8s 2024-03-29T04:10:05.002 INFO:teuthology.orchestra.run.smithi156.stderr: 16100K .......... .......... .......... .......... .......... 9% 20.5M 8s 2024-03-29T04:10:05.004 INFO:teuthology.orchestra.run.smithi156.stderr: 16150K .......... .......... .......... .......... .......... 9% 24.1M 8s 2024-03-29T04:10:05.006 INFO:teuthology.orchestra.run.smithi156.stderr: 16200K .......... .......... .......... .......... .......... 9% 23.9M 8s 2024-03-29T04:10:05.009 INFO:teuthology.orchestra.run.smithi156.stderr: 16250K .......... .......... .......... .......... .......... 9% 24.0M 8s 2024-03-29T04:10:05.011 INFO:teuthology.orchestra.run.smithi156.stderr: 16300K .......... .......... .......... .......... .......... 9% 20.3M 8s 2024-03-29T04:10:05.013 INFO:teuthology.orchestra.run.smithi156.stderr: 16350K .......... .......... .......... .......... .......... 9% 22.3M 8s 2024-03-29T04:10:05.015 INFO:teuthology.orchestra.run.smithi156.stderr: 16400K .......... .......... .......... .......... .......... 9% 30.8M 8s 2024-03-29T04:10:05.017 INFO:teuthology.orchestra.run.smithi156.stderr: 16450K .......... .......... .......... .......... .......... 9% 25.0M 8s 2024-03-29T04:10:05.019 INFO:teuthology.orchestra.run.smithi156.stderr: 16500K .......... .......... .......... .......... .......... 9% 22.0M 8s 2024-03-29T04:10:05.021 INFO:teuthology.orchestra.run.smithi156.stderr: 16550K .......... .......... .......... .......... .......... 9% 26.4M 8s 2024-03-29T04:10:05.022 INFO:teuthology.orchestra.run.smithi156.stderr: 16600K .......... .......... .......... .......... .......... 9% 30.1M 8s 2024-03-29T04:10:05.024 INFO:teuthology.orchestra.run.smithi156.stderr: 16650K .......... .......... .......... .......... .......... 9% 29.3M 8s 2024-03-29T04:10:05.026 INFO:teuthology.orchestra.run.smithi156.stderr: 16700K .......... .......... .......... .......... .......... 9% 25.6M 8s 2024-03-29T04:10:05.028 INFO:teuthology.orchestra.run.smithi156.stderr: 16750K .......... .......... .......... .......... .......... 9% 27.2M 8s 2024-03-29T04:10:05.029 INFO:teuthology.orchestra.run.smithi156.stderr: 16800K .......... .......... .......... .......... .......... 9% 29.7M 8s 2024-03-29T04:10:05.031 INFO:teuthology.orchestra.run.smithi156.stderr: 16850K .......... .......... .......... .......... .......... 10% 29.3M 8s 2024-03-29T04:10:05.033 INFO:teuthology.orchestra.run.smithi156.stderr: 16900K .......... .......... .......... .......... .......... 10% 25.8M 8s 2024-03-29T04:10:05.035 INFO:teuthology.orchestra.run.smithi156.stderr: 16950K .......... .......... .......... .......... .......... 10% 31.1M 8s 2024-03-29T04:10:05.036 INFO:teuthology.orchestra.run.smithi156.stderr: 17000K .......... .......... .......... .......... .......... 10% 31.2M 7s 2024-03-29T04:10:05.038 INFO:teuthology.orchestra.run.smithi156.stderr: 17050K .......... .......... .......... .......... .......... 10% 26.4M 7s 2024-03-29T04:10:05.039 INFO:teuthology.orchestra.run.smithi156.stderr: 17100K .......... .......... .......... .......... .......... 10% 30.1M 7s 2024-03-29T04:10:05.041 INFO:teuthology.orchestra.run.smithi156.stderr: 17150K .......... .......... .......... .......... .......... 10% 34.8M 7s 2024-03-29T04:10:05.043 INFO:teuthology.orchestra.run.smithi156.stderr: 17200K .......... .......... .......... .......... .......... 10% 31.3M 7s 2024-03-29T04:10:05.044 INFO:teuthology.orchestra.run.smithi156.stderr: 17250K .......... .......... .......... .......... .......... 10% 28.5M 7s 2024-03-29T04:10:05.046 INFO:teuthology.orchestra.run.smithi156.stderr: 17300K .......... .......... .......... .......... .......... 10% 22.9M 7s 2024-03-29T04:10:05.048 INFO:teuthology.orchestra.run.smithi156.stderr: 17350K .......... .......... .......... .......... .......... 10% 28.9M 7s 2024-03-29T04:10:05.049 INFO:teuthology.orchestra.run.smithi156.stderr: 17400K .......... .......... .......... .......... .......... 10% 30.6M 7s 2024-03-29T04:10:05.051 INFO:teuthology.orchestra.run.smithi156.stderr: 17450K .......... .......... .......... .......... .......... 10% 29.7M 7s 2024-03-29T04:10:05.053 INFO:teuthology.orchestra.run.smithi156.stderr: 17500K .......... .......... .......... .......... .......... 10% 25.3M 7s 2024-03-29T04:10:05.055 INFO:teuthology.orchestra.run.smithi156.stderr: 17550K .......... .......... .......... .......... .......... 10% 29.6M 7s 2024-03-29T04:10:05.056 INFO:teuthology.orchestra.run.smithi156.stderr: 17600K .......... .......... .......... .......... .......... 10% 29.9M 7s 2024-03-29T04:10:05.058 INFO:teuthology.orchestra.run.smithi156.stderr: 17650K .......... .......... .......... .......... .......... 10% 29.1M 7s 2024-03-29T04:10:05.060 INFO:teuthology.orchestra.run.smithi156.stderr: 17700K .......... .......... .......... .......... .......... 10% 25.2M 7s 2024-03-29T04:10:05.062 INFO:teuthology.orchestra.run.smithi156.stderr: 17750K .......... .......... .......... .......... .......... 10% 29.3M 7s 2024-03-29T04:10:05.064 INFO:teuthology.orchestra.run.smithi156.stderr: 17800K .......... .......... .......... .......... .......... 10% 29.3M 7s 2024-03-29T04:10:05.066 INFO:teuthology.orchestra.run.smithi156.stderr: 17850K .......... .......... .......... .......... .......... 10% 25.3M 7s 2024-03-29T04:10:05.068 INFO:teuthology.orchestra.run.smithi156.stderr: 17900K .......... .......... .......... .......... .......... 10% 21.8M 7s 2024-03-29T04:10:05.069 INFO:teuthology.orchestra.run.smithi156.stderr: 17950K .......... .......... .......... .......... .......... 10% 27.2M 7s 2024-03-29T04:10:05.071 INFO:teuthology.orchestra.run.smithi156.stderr: 18000K .......... .......... .......... .......... .......... 10% 30.4M 7s 2024-03-29T04:10:05.073 INFO:teuthology.orchestra.run.smithi156.stderr: 18050K .......... .......... .......... .......... .......... 10% 31.7M 7s 2024-03-29T04:10:05.075 INFO:teuthology.orchestra.run.smithi156.stderr: 18100K .......... .......... .......... .......... .......... 10% 27.3M 7s 2024-03-29T04:10:05.076 INFO:teuthology.orchestra.run.smithi156.stderr: 18150K .......... .......... .......... .......... .......... 10% 30.8M 7s 2024-03-29T04:10:05.078 INFO:teuthology.orchestra.run.smithi156.stderr: 18200K .......... .......... .......... .......... .......... 10% 31.7M 7s 2024-03-29T04:10:05.080 INFO:teuthology.orchestra.run.smithi156.stderr: 18250K .......... .......... .......... .......... .......... 10% 31.4M 7s 2024-03-29T04:10:05.081 INFO:teuthology.orchestra.run.smithi156.stderr: 18300K .......... .......... .......... .......... .......... 10% 27.0M 7s 2024-03-29T04:10:05.082 INFO:teuthology.orchestra.run.smithi156.stderr: 18350K .......... .......... .......... .......... .......... 10% 31.2M 7s 2024-03-29T04:10:05.084 INFO:teuthology.orchestra.run.smithi156.stderr: 18400K .......... .......... .......... .......... .......... 10% 30.5M 7s 2024-03-29T04:10:05.086 INFO:teuthology.orchestra.run.smithi156.stderr: 18450K .......... .......... .......... .......... .......... 10% 31.5M 7s 2024-03-29T04:10:05.088 INFO:teuthology.orchestra.run.smithi156.stderr: 18500K .......... .......... .......... .......... .......... 11% 25.5M 7s 2024-03-29T04:10:05.089 INFO:teuthology.orchestra.run.smithi156.stderr: 18550K .......... .......... .......... .......... .......... 11% 29.2M 7s 2024-03-29T04:10:05.091 INFO:teuthology.orchestra.run.smithi156.stderr: 18600K .......... .......... .......... .......... .......... 11% 29.9M 7s 2024-03-29T04:10:05.092 INFO:teuthology.orchestra.run.smithi156.stderr: 18650K .......... .......... .......... .......... .......... 11% 29.6M 7s 2024-03-29T04:10:05.094 INFO:teuthology.orchestra.run.smithi156.stderr: 18700K .......... .......... .......... .......... .......... 11% 27.5M 7s 2024-03-29T04:10:05.096 INFO:teuthology.orchestra.run.smithi156.stderr: 18750K .......... .......... .......... .......... .......... 11% 31.4M 7s 2024-03-29T04:10:05.097 INFO:teuthology.orchestra.run.smithi156.stderr: 18800K .......... .......... .......... .......... .......... 11% 30.6M 7s 2024-03-29T04:10:05.099 INFO:teuthology.orchestra.run.smithi156.stderr: 18850K .......... .......... .......... .......... .......... 11% 31.2M 7s 2024-03-29T04:10:05.101 INFO:teuthology.orchestra.run.smithi156.stderr: 18900K .......... .......... .......... .......... .......... 11% 26.9M 7s 2024-03-29T04:10:05.102 INFO:teuthology.orchestra.run.smithi156.stderr: 18950K .......... .......... .......... .......... .......... 11% 30.5M 7s 2024-03-29T04:10:05.104 INFO:teuthology.orchestra.run.smithi156.stderr: 19000K .......... .......... .......... .......... .......... 11% 29.3M 7s 2024-03-29T04:10:05.106 INFO:teuthology.orchestra.run.smithi156.stderr: 19050K .......... .......... .......... .......... .......... 11% 29.7M 7s 2024-03-29T04:10:05.108 INFO:teuthology.orchestra.run.smithi156.stderr: 19100K .......... .......... .......... .......... .......... 11% 25.2M 7s 2024-03-29T04:10:05.109 INFO:teuthology.orchestra.run.smithi156.stderr: 19150K .......... .......... .......... .......... .......... 11% 28.9M 7s 2024-03-29T04:10:05.111 INFO:teuthology.orchestra.run.smithi156.stderr: 19200K .......... .......... .......... .......... .......... 11% 29.2M 7s 2024-03-29T04:10:05.113 INFO:teuthology.orchestra.run.smithi156.stderr: 19250K .......... .......... .......... .......... .......... 11% 29.1M 7s 2024-03-29T04:10:05.115 INFO:teuthology.orchestra.run.smithi156.stderr: 19300K .......... .......... .......... .......... .......... 11% 23.3M 7s 2024-03-29T04:10:05.116 INFO:teuthology.orchestra.run.smithi156.stderr: 19350K .......... .......... .......... .......... .......... 11% 26.4M 7s 2024-03-29T04:10:05.119 INFO:teuthology.orchestra.run.smithi156.stderr: 19400K .......... .......... .......... .......... .......... 11% 26.9M 7s 2024-03-29T04:10:05.120 INFO:teuthology.orchestra.run.smithi156.stderr: 19450K .......... .......... .......... .......... .......... 11% 26.7M 7s 2024-03-29T04:10:05.122 INFO:teuthology.orchestra.run.smithi156.stderr: 19500K .......... .......... .......... .......... .......... 11% 22.2M 7s 2024-03-29T04:10:05.124 INFO:teuthology.orchestra.run.smithi156.stderr: 19550K .......... .......... .......... .......... .......... 11% 25.6M 7s 2024-03-29T04:10:05.126 INFO:teuthology.orchestra.run.smithi156.stderr: 19600K .......... .......... .......... .......... .......... 11% 25.3M 7s 2024-03-29T04:10:05.128 INFO:teuthology.orchestra.run.smithi156.stderr: 19650K .......... .......... .......... .......... .......... 11% 24.8M 7s 2024-03-29T04:10:05.131 INFO:teuthology.orchestra.run.smithi156.stderr: 19700K .......... .......... .......... .......... .......... 11% 21.3M 7s 2024-03-29T04:10:05.132 INFO:teuthology.orchestra.run.smithi156.stderr: 19750K .......... .......... .......... .......... .......... 11% 25.0M 7s 2024-03-29T04:10:05.134 INFO:teuthology.orchestra.run.smithi156.stderr: 19800K .......... .......... .......... .......... .......... 11% 26.9M 7s 2024-03-29T04:10:05.136 INFO:teuthology.orchestra.run.smithi156.stderr: 19850K .......... .......... .......... .......... .......... 11% 27.2M 7s 2024-03-29T04:10:05.138 INFO:teuthology.orchestra.run.smithi156.stderr: 19900K .......... .......... .......... .......... .......... 11% 23.6M 7s 2024-03-29T04:10:05.140 INFO:teuthology.orchestra.run.smithi156.stderr: 19950K .......... .......... .......... .......... .......... 11% 27.0M 7s 2024-03-29T04:10:05.142 INFO:teuthology.orchestra.run.smithi156.stderr: 20000K .......... .......... .......... .......... .......... 11% 27.4M 7s 2024-03-29T04:10:05.144 INFO:teuthology.orchestra.run.smithi156.stderr: 20050K .......... .......... .......... .......... .......... 11% 27.3M 7s 2024-03-29T04:10:05.146 INFO:teuthology.orchestra.run.smithi156.stderr: 20100K .......... .......... .......... .......... .......... 11% 22.8M 7s 2024-03-29T04:10:05.147 INFO:teuthology.orchestra.run.smithi156.stderr: 20150K .......... .......... .......... .......... .......... 11% 26.9M 7s 2024-03-29T04:10:05.149 INFO:teuthology.orchestra.run.smithi156.stderr: 20200K .......... .......... .......... .......... .......... 12% 26.7M 7s 2024-03-29T04:10:05.151 INFO:teuthology.orchestra.run.smithi156.stderr: 20250K .......... .......... .......... .......... .......... 12% 25.6M 7s 2024-03-29T04:10:05.153 INFO:teuthology.orchestra.run.smithi156.stderr: 20300K .......... .......... .......... .......... .......... 12% 22.8M 7s 2024-03-29T04:10:05.156 INFO:teuthology.orchestra.run.smithi156.stderr: 20350K .......... .......... .......... .......... .......... 12% 23.1M 7s 2024-03-29T04:10:05.158 INFO:teuthology.orchestra.run.smithi156.stderr: 20400K .......... .......... .......... .......... .......... 12% 22.8M 7s 2024-03-29T04:10:05.160 INFO:teuthology.orchestra.run.smithi156.stderr: 20450K .......... .......... .......... .......... .......... 12% 21.3M 7s 2024-03-29T04:10:05.162 INFO:teuthology.orchestra.run.smithi156.stderr: 20500K .......... .......... .......... .......... .......... 12% 26.8M 7s 2024-03-29T04:10:05.164 INFO:teuthology.orchestra.run.smithi156.stderr: 20550K .......... .......... .......... .......... .......... 12% 29.1M 7s 2024-03-29T04:10:05.165 INFO:teuthology.orchestra.run.smithi156.stderr: 20600K .......... .......... .......... .......... .......... 12% 28.4M 7s 2024-03-29T04:10:05.167 INFO:teuthology.orchestra.run.smithi156.stderr: 20650K .......... .......... .......... .......... .......... 12% 30.5M 7s 2024-03-29T04:10:05.168 INFO:teuthology.orchestra.run.smithi156.stderr: 20700K .......... .......... .......... .......... .......... 12% 30.2M 7s 2024-03-29T04:10:05.170 INFO:teuthology.orchestra.run.smithi156.stderr: 20750K .......... .......... .......... .......... .......... 12% 35.6M 7s 2024-03-29T04:10:05.171 INFO:teuthology.orchestra.run.smithi156.stderr: 20800K .......... .......... .......... .......... .......... 12% 34.6M 7s 2024-03-29T04:10:05.173 INFO:teuthology.orchestra.run.smithi156.stderr: 20850K .......... .......... .......... .......... .......... 12% 35.4M 7s 2024-03-29T04:10:05.174 INFO:teuthology.orchestra.run.smithi156.stderr: 20900K .......... .......... .......... .......... .......... 12% 30.5M 7s 2024-03-29T04:10:05.176 INFO:teuthology.orchestra.run.smithi156.stderr: 20950K .......... .......... .......... .......... .......... 12% 34.8M 7s 2024-03-29T04:10:05.177 INFO:teuthology.orchestra.run.smithi156.stderr: 21000K .......... .......... .......... .......... .......... 12% 34.5M 7s 2024-03-29T04:10:05.178 INFO:teuthology.orchestra.run.smithi156.stderr: 21050K .......... .......... .......... .......... .......... 12% 34.4M 7s 2024-03-29T04:10:05.180 INFO:teuthology.orchestra.run.smithi156.stderr: 21100K .......... .......... .......... .......... .......... 12% 28.4M 7s 2024-03-29T04:10:05.182 INFO:teuthology.orchestra.run.smithi156.stderr: 21150K .......... .......... .......... .......... .......... 12% 40.1M 7s 2024-03-29T04:10:05.182 INFO:teuthology.orchestra.run.smithi156.stderr: 21200K .......... .......... .......... .......... .......... 12% 40.2M 7s 2024-03-29T04:10:05.184 INFO:teuthology.orchestra.run.smithi156.stderr: 21250K .......... .......... .......... .......... .......... 12% 40.0M 7s 2024-03-29T04:10:05.185 INFO:teuthology.orchestra.run.smithi156.stderr: 21300K .......... .......... .......... .......... .......... 12% 33.0M 7s 2024-03-29T04:10:05.187 INFO:teuthology.orchestra.run.smithi156.stderr: 21350K .......... .......... .......... .......... .......... 12% 39.4M 7s 2024-03-29T04:10:05.188 INFO:teuthology.orchestra.run.smithi156.stderr: 21400K .......... .......... .......... .......... .......... 12% 39.4M 7s 2024-03-29T04:10:05.189 INFO:teuthology.orchestra.run.smithi156.stderr: 21450K .......... .......... .......... .......... .......... 12% 38.6M 7s 2024-03-29T04:10:05.190 INFO:teuthology.orchestra.run.smithi156.stderr: 21500K .......... .......... .......... .......... .......... 12% 32.3M 7s 2024-03-29T04:10:05.192 INFO:teuthology.orchestra.run.smithi156.stderr: 21550K .......... .......... .......... .......... .......... 12% 38.9M 7s 2024-03-29T04:10:05.193 INFO:teuthology.orchestra.run.smithi156.stderr: 21600K .......... .......... .......... .......... .......... 12% 38.1M 7s 2024-03-29T04:10:05.195 INFO:teuthology.orchestra.run.smithi156.stderr: 21650K .......... .......... .......... .......... .......... 12% 38.2M 7s 2024-03-29T04:10:05.196 INFO:teuthology.orchestra.run.smithi156.stderr: 21700K .......... .......... .......... .......... .......... 12% 32.9M 7s 2024-03-29T04:10:05.197 INFO:teuthology.orchestra.run.smithi156.stderr: 21750K .......... .......... .......... .......... .......... 12% 36.5M 7s 2024-03-29T04:10:05.198 INFO:teuthology.orchestra.run.smithi156.stderr: 21800K .......... .......... .......... .......... .......... 12% 36.6M 7s 2024-03-29T04:10:05.200 INFO:teuthology.orchestra.run.smithi156.stderr: 21850K .......... .......... .......... .......... .......... 12% 37.3M 7s 2024-03-29T04:10:05.201 INFO:teuthology.orchestra.run.smithi156.stderr: 21900K .......... .......... .......... .......... .......... 13% 31.4M 7s 2024-03-29T04:10:05.203 INFO:teuthology.orchestra.run.smithi156.stderr: 21950K .......... .......... .......... .......... .......... 13% 36.6M 7s 2024-03-29T04:10:05.204 INFO:teuthology.orchestra.run.smithi156.stderr: 22000K .......... .......... .......... .......... .......... 13% 36.6M 7s 2024-03-29T04:10:05.206 INFO:teuthology.orchestra.run.smithi156.stderr: 22050K .......... .......... .......... .......... .......... 13% 36.4M 7s 2024-03-29T04:10:05.208 INFO:teuthology.orchestra.run.smithi156.stderr: 22100K .......... .......... .......... .......... .......... 13% 31.4M 7s 2024-03-29T04:10:05.208 INFO:teuthology.orchestra.run.smithi156.stderr: 22150K .......... .......... .......... .......... .......... 13% 34.6M 7s 2024-03-29T04:10:05.210 INFO:teuthology.orchestra.run.smithi156.stderr: 22200K .......... .......... .......... .......... .......... 13% 34.1M 7s 2024-03-29T04:10:05.211 INFO:teuthology.orchestra.run.smithi156.stderr: 22250K .......... .......... .......... .......... .......... 13% 32.7M 7s 2024-03-29T04:10:05.213 INFO:teuthology.orchestra.run.smithi156.stderr: 22300K .......... .......... .......... .......... .......... 13% 28.1M 7s 2024-03-29T04:10:05.214 INFO:teuthology.orchestra.run.smithi156.stderr: 22350K .......... .......... .......... .......... .......... 13% 33.2M 7s 2024-03-29T04:10:05.216 INFO:teuthology.orchestra.run.smithi156.stderr: 22400K .......... .......... .......... .......... .......... 13% 33.9M 7s 2024-03-29T04:10:05.217 INFO:teuthology.orchestra.run.smithi156.stderr: 22450K .......... .......... .......... .......... .......... 13% 35.8M 7s 2024-03-29T04:10:05.219 INFO:teuthology.orchestra.run.smithi156.stderr: 22500K .......... .......... .......... .......... .......... 13% 29.9M 7s 2024-03-29T04:10:05.220 INFO:teuthology.orchestra.run.smithi156.stderr: 22550K .......... .......... .......... .......... .......... 13% 34.6M 7s 2024-03-29T04:10:05.222 INFO:teuthology.orchestra.run.smithi156.stderr: 22600K .......... .......... .......... .......... .......... 13% 28.4M 7s 2024-03-29T04:10:05.224 INFO:teuthology.orchestra.run.smithi156.stderr: 22650K .......... .......... .......... .......... .......... 13% 33.0M 7s 2024-03-29T04:10:05.225 INFO:teuthology.orchestra.run.smithi156.stderr: 22700K .......... .......... .......... .......... .......... 13% 28.9M 7s 2024-03-29T04:10:05.227 INFO:teuthology.orchestra.run.smithi156.stderr: 22750K .......... .......... .......... .......... .......... 13% 34.2M 7s 2024-03-29T04:10:05.228 INFO:teuthology.orchestra.run.smithi156.stderr: 22800K .......... .......... .......... .......... .......... 13% 33.7M 7s 2024-03-29T04:10:05.229 INFO:teuthology.orchestra.run.smithi156.stderr: 22850K .......... .......... .......... .......... .......... 13% 33.1M 7s 2024-03-29T04:10:05.231 INFO:teuthology.orchestra.run.smithi156.stderr: 22900K .......... .......... .......... .......... .......... 13% 27.7M 7s 2024-03-29T04:10:05.233 INFO:teuthology.orchestra.run.smithi156.stderr: 22950K .......... .......... .......... .......... .......... 13% 30.8M 7s 2024-03-29T04:10:05.234 INFO:teuthology.orchestra.run.smithi156.stderr: 23000K .......... .......... .......... .......... .......... 13% 30.1M 7s 2024-03-29T04:10:05.236 INFO:teuthology.orchestra.run.smithi156.stderr: 23050K .......... .......... .......... .......... .......... 13% 28.2M 7s 2024-03-29T04:10:05.238 INFO:teuthology.orchestra.run.smithi156.stderr: 23100K .......... .......... .......... .......... .......... 13% 24.1M 7s 2024-03-29T04:10:05.240 INFO:teuthology.orchestra.run.smithi156.stderr: 23150K .......... .......... .......... .......... .......... 13% 31.0M 7s 2024-03-29T04:10:05.241 INFO:teuthology.orchestra.run.smithi156.stderr: 23200K .......... .......... .......... .......... .......... 13% 31.5M 7s 2024-03-29T04:10:05.243 INFO:teuthology.orchestra.run.smithi156.stderr: 23250K .......... .......... .......... .......... .......... 13% 31.7M 7s 2024-03-29T04:10:05.245 INFO:teuthology.orchestra.run.smithi156.stderr: 23300K .......... .......... .......... .......... .......... 13% 27.1M 7s 2024-03-29T04:10:05.246 INFO:teuthology.orchestra.run.smithi156.stderr: 23350K .......... .......... .......... .......... .......... 13% 31.4M 7s 2024-03-29T04:10:05.248 INFO:teuthology.orchestra.run.smithi156.stderr: 23400K .......... .......... .......... .......... .......... 13% 31.3M 7s 2024-03-29T04:10:05.250 INFO:teuthology.orchestra.run.smithi156.stderr: 23450K .......... .......... .......... .......... .......... 13% 31.1M 7s 2024-03-29T04:10:05.251 INFO:teuthology.orchestra.run.smithi156.stderr: 23500K .......... .......... .......... .......... .......... 13% 25.9M 7s 2024-03-29T04:10:05.253 INFO:teuthology.orchestra.run.smithi156.stderr: 23550K .......... .......... .......... .......... .......... 14% 30.4M 7s 2024-03-29T04:10:05.254 INFO:teuthology.orchestra.run.smithi156.stderr: 23600K .......... .......... .......... .......... .......... 14% 30.8M 7s 2024-03-29T04:10:05.256 INFO:teuthology.orchestra.run.smithi156.stderr: 23650K .......... .......... .......... .......... .......... 14% 29.0M 6s 2024-03-29T04:10:05.259 INFO:teuthology.orchestra.run.smithi156.stderr: 23700K .......... .......... .......... .......... .......... 14% 19.0M 6s 2024-03-29T04:10:05.261 INFO:teuthology.orchestra.run.smithi156.stderr: 23750K .......... .......... .......... .......... .......... 14% 23.6M 6s 2024-03-29T04:10:05.263 INFO:teuthology.orchestra.run.smithi156.stderr: 23800K .......... .......... .......... .......... .......... 14% 23.9M 6s 2024-03-29T04:10:05.265 INFO:teuthology.orchestra.run.smithi156.stderr: 23850K .......... .......... .......... .......... .......... 14% 26.4M 6s 2024-03-29T04:10:05.267 INFO:teuthology.orchestra.run.smithi156.stderr: 23900K .......... .......... .......... .......... .......... 14% 24.3M 6s 2024-03-29T04:10:05.269 INFO:teuthology.orchestra.run.smithi156.stderr: 23950K .......... .......... .......... .......... .......... 14% 28.5M 6s 2024-03-29T04:10:05.270 INFO:teuthology.orchestra.run.smithi156.stderr: 24000K .......... .......... .......... .......... .......... 14% 28.3M 6s 2024-03-29T04:10:05.272 INFO:teuthology.orchestra.run.smithi156.stderr: 24050K .......... .......... .......... .......... .......... 14% 28.2M 6s 2024-03-29T04:10:05.274 INFO:teuthology.orchestra.run.smithi156.stderr: 24100K .......... .......... .......... .......... .......... 14% 23.9M 6s 2024-03-29T04:10:05.276 INFO:teuthology.orchestra.run.smithi156.stderr: 24150K .......... .......... .......... .......... .......... 14% 26.0M 6s 2024-03-29T04:10:05.278 INFO:teuthology.orchestra.run.smithi156.stderr: 24200K .......... .......... .......... .......... .......... 14% 25.1M 6s 2024-03-29T04:10:05.280 INFO:teuthology.orchestra.run.smithi156.stderr: 24250K .......... .......... .......... .......... .......... 14% 22.7M 6s 2024-03-29T04:10:05.282 INFO:teuthology.orchestra.run.smithi156.stderr: 24300K .......... .......... .......... .......... .......... 14% 20.2M 6s 2024-03-29T04:10:05.284 INFO:teuthology.orchestra.run.smithi156.stderr: 24350K .......... .......... .......... .......... .......... 14% 23.8M 6s 2024-03-29T04:10:05.286 INFO:teuthology.orchestra.run.smithi156.stderr: 24400K .......... .......... .......... .......... .......... 14% 23.4M 6s 2024-03-29T04:10:05.289 INFO:teuthology.orchestra.run.smithi156.stderr: 24450K .......... .......... .......... .......... .......... 14% 23.0M 6s 2024-03-29T04:10:05.291 INFO:teuthology.orchestra.run.smithi156.stderr: 24500K .......... .......... .......... .......... .......... 14% 20.1M 6s 2024-03-29T04:10:05.293 INFO:teuthology.orchestra.run.smithi156.stderr: 24550K .......... .......... .......... .......... .......... 14% 23.0M 6s 2024-03-29T04:10:05.301 INFO:teuthology.orchestra.run.smithi156.stderr: 24600K .......... .......... .......... .......... .......... 14% 30.3M 6s 2024-03-29T04:10:05.301 INFO:teuthology.orchestra.run.smithi156.stderr: 24650K .......... .......... .......... .......... .......... 14% 26.9M 6s 2024-03-29T04:10:05.301 INFO:teuthology.orchestra.run.smithi156.stderr: 24700K .......... .......... .......... .......... .......... 14% 28.8M 6s 2024-03-29T04:10:05.301 INFO:teuthology.orchestra.run.smithi156.stderr: 24750K .......... .......... .......... .......... .......... 14% 38.1M 6s 2024-03-29T04:10:05.301 INFO:teuthology.orchestra.run.smithi156.stderr: 24800K .......... .......... .......... .......... .......... 14% 39.2M 6s 2024-03-29T04:10:05.302 INFO:teuthology.orchestra.run.smithi156.stderr: 24850K .......... .......... .......... .......... .......... 14% 34.0M 6s 2024-03-29T04:10:05.304 INFO:teuthology.orchestra.run.smithi156.stderr: 24900K .......... .......... .......... .......... .......... 14% 27.9M 6s 2024-03-29T04:10:05.305 INFO:teuthology.orchestra.run.smithi156.stderr: 24950K .......... .......... .......... .......... .......... 14% 35.8M 6s 2024-03-29T04:10:05.307 INFO:teuthology.orchestra.run.smithi156.stderr: 25000K .......... .......... .......... .......... .......... 14% 37.1M 6s 2024-03-29T04:10:05.309 INFO:teuthology.orchestra.run.smithi156.stderr: 25050K .......... .......... .......... .......... .......... 14% 31.6M 6s 2024-03-29T04:10:05.310 INFO:teuthology.orchestra.run.smithi156.stderr: 25100K .......... .......... .......... .......... .......... 14% 29.0M 6s 2024-03-29T04:10:05.312 INFO:teuthology.orchestra.run.smithi156.stderr: 25150K .......... .......... .......... .......... .......... 14% 33.4M 6s 2024-03-29T04:10:05.313 INFO:teuthology.orchestra.run.smithi156.stderr: 25200K .......... .......... .......... .......... .......... 14% 35.1M 6s 2024-03-29T04:10:05.315 INFO:teuthology.orchestra.run.smithi156.stderr: 25250K .......... .......... .......... .......... .......... 15% 35.3M 6s 2024-03-29T04:10:05.316 INFO:teuthology.orchestra.run.smithi156.stderr: 25300K .......... .......... .......... .......... .......... 15% 30.0M 6s 2024-03-29T04:10:05.318 INFO:teuthology.orchestra.run.smithi156.stderr: 25350K .......... .......... .......... .......... .......... 15% 34.8M 6s 2024-03-29T04:10:05.319 INFO:teuthology.orchestra.run.smithi156.stderr: 25400K .......... .......... .......... .......... .......... 15% 35.3M 6s 2024-03-29T04:10:05.320 INFO:teuthology.orchestra.run.smithi156.stderr: 25450K .......... .......... .......... .......... .......... 15% 34.3M 6s 2024-03-29T04:10:05.321 INFO:teuthology.orchestra.run.smithi156.stderr: 25500K .......... .......... .......... .......... .......... 15% 31.2M 6s 2024-03-29T04:10:05.323 INFO:teuthology.orchestra.run.smithi156.stderr: 25550K .......... .......... .......... .......... .......... 15% 36.6M 6s 2024-03-29T04:10:05.324 INFO:teuthology.orchestra.run.smithi156.stderr: 25600K .......... .......... .......... .......... .......... 15% 36.7M 6s 2024-03-29T04:10:05.396 INFO:teuthology.orchestra.run.smithi156.stderr: 25650K .......... .......... .......... .......... .......... 15% 703K 7s 2024-03-29T04:10:05.397 INFO:teuthology.orchestra.run.smithi156.stderr: 25700K .......... .......... .......... .......... .......... 15% 30.1M 7s 2024-03-29T04:10:05.398 INFO:teuthology.orchestra.run.smithi156.stderr: 25750K .......... .......... .......... .......... .......... 15% 36.6M 7s 2024-03-29T04:10:05.400 INFO:teuthology.orchestra.run.smithi156.stderr: 25800K .......... .......... .......... .......... .......... 15% 29.2M 7s 2024-03-29T04:10:05.402 INFO:teuthology.orchestra.run.smithi156.stderr: 25850K .......... .......... .......... .......... .......... 15% 28.9M 7s 2024-03-29T04:10:05.403 INFO:teuthology.orchestra.run.smithi156.stderr: 25900K .......... .......... .......... .......... .......... 15% 27.5M 7s 2024-03-29T04:10:05.405 INFO:teuthology.orchestra.run.smithi156.stderr: 25950K .......... .......... .......... .......... .......... 15% 38.1M 7s 2024-03-29T04:10:05.406 INFO:teuthology.orchestra.run.smithi156.stderr: 26000K .......... .......... .......... .......... .......... 15% 34.4M 7s 2024-03-29T04:10:05.407 INFO:teuthology.orchestra.run.smithi156.stderr: 26050K .......... .......... .......... .......... .......... 15% 35.9M 7s 2024-03-29T04:10:05.409 INFO:teuthology.orchestra.run.smithi156.stderr: 26100K .......... .......... .......... .......... .......... 15% 31.0M 7s 2024-03-29T04:10:05.410 INFO:teuthology.orchestra.run.smithi156.stderr: 26150K .......... .......... .......... .......... .......... 15% 36.2M 7s 2024-03-29T04:10:05.412 INFO:teuthology.orchestra.run.smithi156.stderr: 26200K .......... .......... .......... .......... .......... 15% 36.1M 7s 2024-03-29T04:10:05.413 INFO:teuthology.orchestra.run.smithi156.stderr: 26250K .......... .......... .......... .......... .......... 15% 36.2M 7s 2024-03-29T04:10:05.415 INFO:teuthology.orchestra.run.smithi156.stderr: 26300K .......... .......... .......... .......... .......... 15% 30.8M 7s 2024-03-29T04:10:05.416 INFO:teuthology.orchestra.run.smithi156.stderr: 26350K .......... .......... .......... .......... .......... 15% 34.7M 7s 2024-03-29T04:10:05.418 INFO:teuthology.orchestra.run.smithi156.stderr: 26400K .......... .......... .......... .......... .......... 15% 35.5M 7s 2024-03-29T04:10:05.419 INFO:teuthology.orchestra.run.smithi156.stderr: 26450K .......... .......... .......... .......... .......... 15% 35.9M 7s 2024-03-29T04:10:05.421 INFO:teuthology.orchestra.run.smithi156.stderr: 26500K .......... .......... .......... .......... .......... 15% 28.3M 7s 2024-03-29T04:10:05.423 INFO:teuthology.orchestra.run.smithi156.stderr: 26550K .......... .......... .......... .......... .......... 15% 29.0M 7s 2024-03-29T04:10:05.425 INFO:teuthology.orchestra.run.smithi156.stderr: 26600K .......... .......... .......... .......... .......... 15% 28.9M 7s 2024-03-29T04:10:05.427 INFO:teuthology.orchestra.run.smithi156.stderr: 26650K .......... .......... .......... .......... .......... 15% 29.2M 7s 2024-03-29T04:10:05.428 INFO:teuthology.orchestra.run.smithi156.stderr: 26700K .......... .......... .......... .......... .......... 15% 24.7M 7s 2024-03-29T04:10:05.429 INFO:teuthology.orchestra.run.smithi156.stderr: 26750K .......... .......... .......... .......... .......... 15% 29.9M 7s 2024-03-29T04:10:05.431 INFO:teuthology.orchestra.run.smithi156.stderr: 26800K .......... .......... .......... .......... .......... 15% 33.2M 7s 2024-03-29T04:10:05.432 INFO:teuthology.orchestra.run.smithi156.stderr: 26850K .......... .......... .......... .......... .......... 15% 33.4M 7s 2024-03-29T04:10:05.434 INFO:teuthology.orchestra.run.smithi156.stderr: 26900K .......... .......... .......... .......... .......... 15% 28.0M 7s 2024-03-29T04:10:05.436 INFO:teuthology.orchestra.run.smithi156.stderr: 26950K .......... .......... .......... .......... .......... 16% 27.7M 7s 2024-03-29T04:10:05.438 INFO:teuthology.orchestra.run.smithi156.stderr: 27000K .......... .......... .......... .......... .......... 16% 28.3M 7s 2024-03-29T04:10:05.439 INFO:teuthology.orchestra.run.smithi156.stderr: 27050K .......... .......... .......... .......... .......... 16% 27.3M 7s 2024-03-29T04:10:05.441 INFO:teuthology.orchestra.run.smithi156.stderr: 27100K .......... .......... .......... .......... .......... 16% 22.9M 7s 2024-03-29T04:10:05.444 INFO:teuthology.orchestra.run.smithi156.stderr: 27150K .......... .......... .......... .......... .......... 16% 27.7M 6s 2024-03-29T04:10:05.445 INFO:teuthology.orchestra.run.smithi156.stderr: 27200K .......... .......... .......... .......... .......... 16% 28.7M 6s 2024-03-29T04:10:05.447 INFO:teuthology.orchestra.run.smithi156.stderr: 27250K .......... .......... .......... .......... .......... 16% 29.1M 6s 2024-03-29T04:10:05.448 INFO:teuthology.orchestra.run.smithi156.stderr: 27300K .......... .......... .......... .......... .......... 16% 24.8M 6s 2024-03-29T04:10:05.450 INFO:teuthology.orchestra.run.smithi156.stderr: 27350K .......... .......... .......... .......... .......... 16% 29.0M 6s 2024-03-29T04:10:05.452 INFO:teuthology.orchestra.run.smithi156.stderr: 27400K .......... .......... .......... .......... .......... 16% 28.8M 6s 2024-03-29T04:10:05.453 INFO:teuthology.orchestra.run.smithi156.stderr: 27450K .......... .......... .......... .......... .......... 16% 32.7M 6s 2024-03-29T04:10:05.455 INFO:teuthology.orchestra.run.smithi156.stderr: 27500K .......... .......... .......... .......... .......... 16% 29.3M 6s 2024-03-29T04:10:05.456 INFO:teuthology.orchestra.run.smithi156.stderr: 27550K .......... .......... .......... .......... .......... 16% 32.6M 6s 2024-03-29T04:10:05.458 INFO:teuthology.orchestra.run.smithi156.stderr: 27600K .......... .......... .......... .......... .......... 16% 32.7M 6s 2024-03-29T04:10:05.460 INFO:teuthology.orchestra.run.smithi156.stderr: 27650K .......... .......... .......... .......... .......... 16% 32.2M 6s 2024-03-29T04:10:05.461 INFO:teuthology.orchestra.run.smithi156.stderr: 27700K .......... .......... .......... .......... .......... 16% 27.9M 6s 2024-03-29T04:10:05.463 INFO:teuthology.orchestra.run.smithi156.stderr: 27750K .......... .......... .......... .......... .......... 16% 31.6M 6s 2024-03-29T04:10:05.464 INFO:teuthology.orchestra.run.smithi156.stderr: 27800K .......... .......... .......... .......... .......... 16% 30.5M 6s 2024-03-29T04:10:05.466 INFO:teuthology.orchestra.run.smithi156.stderr: 27850K .......... .......... .......... .......... .......... 16% 31.1M 6s 2024-03-29T04:10:05.468 INFO:teuthology.orchestra.run.smithi156.stderr: 27900K .......... .......... .......... .......... .......... 16% 26.7M 6s 2024-03-29T04:10:05.469 INFO:teuthology.orchestra.run.smithi156.stderr: 27950K .......... .......... .......... .......... .......... 16% 30.9M 6s 2024-03-29T04:10:05.471 INFO:teuthology.orchestra.run.smithi156.stderr: 28000K .......... .......... .......... .......... .......... 16% 24.5M 6s 2024-03-29T04:10:05.474 INFO:teuthology.orchestra.run.smithi156.stderr: 28050K .......... .......... .......... .......... .......... 16% 23.3M 6s 2024-03-29T04:10:05.476 INFO:teuthology.orchestra.run.smithi156.stderr: 28100K .......... .......... .......... .......... .......... 16% 19.5M 6s 2024-03-29T04:10:05.478 INFO:teuthology.orchestra.run.smithi156.stderr: 28150K .......... .......... .......... .......... .......... 16% 22.7M 6s 2024-03-29T04:10:05.480 INFO:teuthology.orchestra.run.smithi156.stderr: 28200K .......... .......... .......... .......... .......... 16% 22.6M 6s 2024-03-29T04:10:05.483 INFO:teuthology.orchestra.run.smithi156.stderr: 28250K .......... .......... .......... .......... .......... 16% 22.9M 6s 2024-03-29T04:10:05.485 INFO:teuthology.orchestra.run.smithi156.stderr: 28300K .......... .......... .......... .......... .......... 16% 19.4M 6s 2024-03-29T04:10:05.487 INFO:teuthology.orchestra.run.smithi156.stderr: 28350K .......... .......... .......... .......... .......... 16% 22.4M 6s 2024-03-29T04:10:05.489 INFO:teuthology.orchestra.run.smithi156.stderr: 28400K .......... .......... .......... .......... .......... 16% 22.5M 6s 2024-03-29T04:10:05.492 INFO:teuthology.orchestra.run.smithi156.stderr: 28450K .......... .......... .......... .......... .......... 16% 22.7M 6s 2024-03-29T04:10:05.494 INFO:teuthology.orchestra.run.smithi156.stderr: 28500K .......... .......... .......... .......... .......... 16% 19.3M 6s 2024-03-29T04:10:05.496 INFO:teuthology.orchestra.run.smithi156.stderr: 28550K .......... .......... .......... .......... .......... 16% 22.7M 6s 2024-03-29T04:10:05.498 INFO:teuthology.orchestra.run.smithi156.stderr: 28600K .......... .......... .......... .......... .......... 16% 22.1M 6s 2024-03-29T04:10:05.502 INFO:teuthology.orchestra.run.smithi156.stderr: 28650K .......... .......... .......... .......... .......... 17% 22.2M 6s 2024-03-29T04:10:05.503 INFO:teuthology.orchestra.run.smithi156.stderr: 28700K .......... .......... .......... .......... .......... 17% 22.0M 6s 2024-03-29T04:10:05.504 INFO:teuthology.orchestra.run.smithi156.stderr: 28750K .......... .......... .......... .......... .......... 17% 34.4M 6s 2024-03-29T04:10:05.506 INFO:teuthology.orchestra.run.smithi156.stderr: 28800K .......... .......... .......... .......... .......... 17% 34.1M 6s 2024-03-29T04:10:05.507 INFO:teuthology.orchestra.run.smithi156.stderr: 28850K .......... .......... .......... .......... .......... 17% 36.7M 6s 2024-03-29T04:10:05.509 INFO:teuthology.orchestra.run.smithi156.stderr: 28900K .......... .......... .......... .......... .......... 17% 26.4M 6s 2024-03-29T04:10:05.510 INFO:teuthology.orchestra.run.smithi156.stderr: 28950K .......... .......... .......... .......... .......... 17% 33.3M 6s 2024-03-29T04:10:05.512 INFO:teuthology.orchestra.run.smithi156.stderr: 29000K .......... .......... .......... .......... .......... 17% 33.9M 6s 2024-03-29T04:10:05.513 INFO:teuthology.orchestra.run.smithi156.stderr: 29050K .......... .......... .......... .......... .......... 17% 33.4M 6s 2024-03-29T04:10:05.515 INFO:teuthology.orchestra.run.smithi156.stderr: 29100K .......... .......... .......... .......... .......... 17% 28.5M 6s 2024-03-29T04:10:05.516 INFO:teuthology.orchestra.run.smithi156.stderr: 29150K .......... .......... .......... .......... .......... 17% 33.4M 6s 2024-03-29T04:10:05.518 INFO:teuthology.orchestra.run.smithi156.stderr: 29200K .......... .......... .......... .......... .......... 17% 33.8M 6s 2024-03-29T04:10:05.519 INFO:teuthology.orchestra.run.smithi156.stderr: 29250K .......... .......... .......... .......... .......... 17% 32.8M 6s 2024-03-29T04:10:05.521 INFO:teuthology.orchestra.run.smithi156.stderr: 29300K .......... .......... .......... .......... .......... 17% 27.6M 6s 2024-03-29T04:10:05.522 INFO:teuthology.orchestra.run.smithi156.stderr: 29350K .......... .......... .......... .......... .......... 17% 31.8M 6s 2024-03-29T04:10:05.524 INFO:teuthology.orchestra.run.smithi156.stderr: 29400K .......... .......... .......... .......... .......... 17% 32.6M 6s 2024-03-29T04:10:05.525 INFO:teuthology.orchestra.run.smithi156.stderr: 29450K .......... .......... .......... .......... .......... 17% 34.2M 6s 2024-03-29T04:10:05.527 INFO:teuthology.orchestra.run.smithi156.stderr: 29500K .......... .......... .......... .......... .......... 17% 26.1M 6s 2024-03-29T04:10:05.529 INFO:teuthology.orchestra.run.smithi156.stderr: 29550K .......... .......... .......... .......... .......... 17% 30.5M 6s 2024-03-29T04:10:05.530 INFO:teuthology.orchestra.run.smithi156.stderr: 29600K .......... .......... .......... .......... .......... 17% 30.5M 6s 2024-03-29T04:10:05.532 INFO:teuthology.orchestra.run.smithi156.stderr: 29650K .......... .......... .......... .......... .......... 17% 32.6M 6s 2024-03-29T04:10:05.534 INFO:teuthology.orchestra.run.smithi156.stderr: 29700K .......... .......... .......... .......... .......... 17% 29.2M 6s 2024-03-29T04:10:05.535 INFO:teuthology.orchestra.run.smithi156.stderr: 29750K .......... .......... .......... .......... .......... 17% 34.0M 6s 2024-03-29T04:10:05.536 INFO:teuthology.orchestra.run.smithi156.stderr: 29800K .......... .......... .......... .......... .......... 17% 33.7M 6s 2024-03-29T04:10:05.538 INFO:teuthology.orchestra.run.smithi156.stderr: 29850K .......... .......... .......... .......... .......... 17% 34.0M 6s 2024-03-29T04:10:05.540 INFO:teuthology.orchestra.run.smithi156.stderr: 29900K .......... .......... .......... .......... .......... 17% 29.2M 6s 2024-03-29T04:10:05.541 INFO:teuthology.orchestra.run.smithi156.stderr: 29950K .......... .......... .......... .......... .......... 17% 27.4M 6s 2024-03-29T04:10:05.543 INFO:teuthology.orchestra.run.smithi156.stderr: 30000K .......... .......... .......... .......... .......... 17% 27.3M 6s 2024-03-29T04:10:05.545 INFO:teuthology.orchestra.run.smithi156.stderr: 30050K .......... .......... .......... .......... .......... 17% 27.8M 6s 2024-03-29T04:10:05.547 INFO:teuthology.orchestra.run.smithi156.stderr: 30100K .......... .......... .......... .......... .......... 17% 24.5M 6s 2024-03-29T04:10:05.549 INFO:teuthology.orchestra.run.smithi156.stderr: 30150K .......... .......... .......... .......... .......... 17% 28.5M 6s 2024-03-29T04:10:05.551 INFO:teuthology.orchestra.run.smithi156.stderr: 30200K .......... .......... .......... .......... .......... 17% 28.1M 6s 2024-03-29T04:10:05.552 INFO:teuthology.orchestra.run.smithi156.stderr: 30250K .......... .......... .......... .......... .......... 17% 27.8M 6s 2024-03-29T04:10:05.554 INFO:teuthology.orchestra.run.smithi156.stderr: 30300K .......... .......... .......... .......... .......... 18% 24.1M 6s 2024-03-29T04:10:05.556 INFO:teuthology.orchestra.run.smithi156.stderr: 30350K .......... .......... .......... .......... .......... 18% 27.7M 6s 2024-03-29T04:10:05.558 INFO:teuthology.orchestra.run.smithi156.stderr: 30400K .......... .......... .......... .......... .......... 18% 27.8M 6s 2024-03-29T04:10:05.560 INFO:teuthology.orchestra.run.smithi156.stderr: 30450K .......... .......... .......... .......... .......... 18% 28.8M 6s 2024-03-29T04:10:05.561 INFO:teuthology.orchestra.run.smithi156.stderr: 30500K .......... .......... .......... .......... .......... 18% 32.2M 6s 2024-03-29T04:10:05.562 INFO:teuthology.orchestra.run.smithi156.stderr: 30550K .......... .......... .......... .......... .......... 18% 38.1M 6s 2024-03-29T04:10:05.563 INFO:teuthology.orchestra.run.smithi156.stderr: 30600K .......... .......... .......... .......... .......... 18% 37.7M 6s 2024-03-29T04:10:05.565 INFO:teuthology.orchestra.run.smithi156.stderr: 30650K .......... .......... .......... .......... .......... 18% 38.3M 6s 2024-03-29T04:10:05.566 INFO:teuthology.orchestra.run.smithi156.stderr: 30700K .......... .......... .......... .......... .......... 18% 32.5M 6s 2024-03-29T04:10:05.568 INFO:teuthology.orchestra.run.smithi156.stderr: 30750K .......... .......... .......... .......... .......... 18% 37.5M 6s 2024-03-29T04:10:05.569 INFO:teuthology.orchestra.run.smithi156.stderr: 30800K .......... .......... .......... .......... .......... 18% 37.7M 6s 2024-03-29T04:10:05.570 INFO:teuthology.orchestra.run.smithi156.stderr: 30850K .......... .......... .......... .......... .......... 18% 36.7M 6s 2024-03-29T04:10:05.572 INFO:teuthology.orchestra.run.smithi156.stderr: 30900K .......... .......... .......... .......... .......... 18% 26.5M 6s 2024-03-29T04:10:05.574 INFO:teuthology.orchestra.run.smithi156.stderr: 30950K .......... .......... .......... .......... .......... 18% 30.9M 6s 2024-03-29T04:10:05.575 INFO:teuthology.orchestra.run.smithi156.stderr: 31000K .......... .......... .......... .......... .......... 18% 30.9M 6s 2024-03-29T04:10:05.577 INFO:teuthology.orchestra.run.smithi156.stderr: 31050K .......... .......... .......... .......... .......... 18% 30.7M 6s 2024-03-29T04:10:05.579 INFO:teuthology.orchestra.run.smithi156.stderr: 31100K .......... .......... .......... .......... .......... 18% 26.8M 6s 2024-03-29T04:10:05.580 INFO:teuthology.orchestra.run.smithi156.stderr: 31150K .......... .......... .......... .......... .......... 18% 30.2M 6s 2024-03-29T04:10:05.582 INFO:teuthology.orchestra.run.smithi156.stderr: 31200K .......... .......... .......... .......... .......... 18% 30.3M 6s 2024-03-29T04:10:05.584 INFO:teuthology.orchestra.run.smithi156.stderr: 31250K .......... .......... .......... .......... .......... 18% 30.4M 6s 2024-03-29T04:10:05.585 INFO:teuthology.orchestra.run.smithi156.stderr: 31300K .......... .......... .......... .......... .......... 18% 26.2M 6s 2024-03-29T04:10:05.587 INFO:teuthology.orchestra.run.smithi156.stderr: 31350K .......... .......... .......... .......... .......... 18% 30.1M 6s 2024-03-29T04:10:05.588 INFO:teuthology.orchestra.run.smithi156.stderr: 31400K .......... .......... .......... .......... .......... 18% 29.8M 6s 2024-03-29T04:10:05.590 INFO:teuthology.orchestra.run.smithi156.stderr: 31450K .......... .......... .......... .......... .......... 18% 29.2M 6s 2024-03-29T04:10:05.593 INFO:teuthology.orchestra.run.smithi156.stderr: 31500K .......... .......... .......... .......... .......... 18% 24.9M 6s 2024-03-29T04:10:05.594 INFO:teuthology.orchestra.run.smithi156.stderr: 31550K .......... .......... .......... .......... .......... 18% 28.6M 6s 2024-03-29T04:10:05.596 INFO:teuthology.orchestra.run.smithi156.stderr: 31600K .......... .......... .......... .......... .......... 18% 22.8M 6s 2024-03-29T04:10:05.599 INFO:teuthology.orchestra.run.smithi156.stderr: 31650K .......... .......... .......... .......... .......... 18% 23.0M 6s 2024-03-29T04:10:05.601 INFO:teuthology.orchestra.run.smithi156.stderr: 31700K .......... .......... .......... .......... .......... 18% 19.3M 6s 2024-03-29T04:10:05.603 INFO:teuthology.orchestra.run.smithi156.stderr: 31750K .......... .......... .......... .......... .......... 18% 22.2M 6s 2024-03-29T04:10:05.606 INFO:teuthology.orchestra.run.smithi156.stderr: 31800K .......... .......... .......... .......... .......... 18% 21.8M 6s 2024-03-29T04:10:05.608 INFO:teuthology.orchestra.run.smithi156.stderr: 31850K .......... .......... .......... .......... .......... 18% 22.6M 6s 2024-03-29T04:10:05.610 INFO:teuthology.orchestra.run.smithi156.stderr: 31900K .......... .......... .......... .......... .......... 18% 19.4M 6s 2024-03-29T04:10:05.612 INFO:teuthology.orchestra.run.smithi156.stderr: 31950K .......... .......... .......... .......... .......... 18% 22.4M 6s 2024-03-29T04:10:05.614 INFO:teuthology.orchestra.run.smithi156.stderr: 32000K .......... .......... .......... .......... .......... 19% 22.2M 6s 2024-03-29T04:10:05.617 INFO:teuthology.orchestra.run.smithi156.stderr: 32050K .......... .......... .......... .......... .......... 19% 22.2M 6s 2024-03-29T04:10:05.619 INFO:teuthology.orchestra.run.smithi156.stderr: 32100K .......... .......... .......... .......... .......... 19% 18.7M 6s 2024-03-29T04:10:05.621 INFO:teuthology.orchestra.run.smithi156.stderr: 32150K .......... .......... .......... .......... .......... 19% 22.0M 6s 2024-03-29T04:10:05.624 INFO:teuthology.orchestra.run.smithi156.stderr: 32200K .......... .......... .......... .......... .......... 19% 21.8M 6s 2024-03-29T04:10:05.626 INFO:teuthology.orchestra.run.smithi156.stderr: 32250K .......... .......... .......... .......... .......... 19% 21.8M 6s 2024-03-29T04:10:05.628 INFO:teuthology.orchestra.run.smithi156.stderr: 32300K .......... .......... .......... .......... .......... 19% 18.2M 6s 2024-03-29T04:10:05.631 INFO:teuthology.orchestra.run.smithi156.stderr: 32350K .......... .......... .......... .......... .......... 19% 21.6M 6s 2024-03-29T04:10:05.633 INFO:teuthology.orchestra.run.smithi156.stderr: 32400K .......... .......... .......... .......... .......... 19% 21.6M 6s 2024-03-29T04:10:05.636 INFO:teuthology.orchestra.run.smithi156.stderr: 32450K .......... .......... .......... .......... .......... 19% 21.7M 6s 2024-03-29T04:10:05.637 INFO:teuthology.orchestra.run.smithi156.stderr: 32500K .......... .......... .......... .......... .......... 19% 21.7M 6s 2024-03-29T04:10:05.639 INFO:teuthology.orchestra.run.smithi156.stderr: 32550K .......... .......... .......... .......... .......... 19% 26.9M 6s 2024-03-29T04:10:05.642 INFO:teuthology.orchestra.run.smithi156.stderr: 32600K .......... .......... .......... .......... .......... 19% 25.2M 6s 2024-03-29T04:10:05.643 INFO:teuthology.orchestra.run.smithi156.stderr: 32650K .......... .......... .......... .......... .......... 19% 25.0M 6s 2024-03-29T04:10:05.646 INFO:teuthology.orchestra.run.smithi156.stderr: 32700K .......... .......... .......... .......... .......... 19% 21.0M 6s 2024-03-29T04:10:05.648 INFO:teuthology.orchestra.run.smithi156.stderr: 32750K .......... .......... .......... .......... .......... 19% 23.0M 6s 2024-03-29T04:10:05.649 INFO:teuthology.orchestra.run.smithi156.stderr: 32800K .......... .......... .......... .......... .......... 19% 30.4M 6s 2024-03-29T04:10:05.652 INFO:teuthology.orchestra.run.smithi156.stderr: 32850K .......... .......... .......... .......... .......... 19% 31.9M 6s 2024-03-29T04:10:05.653 INFO:teuthology.orchestra.run.smithi156.stderr: 32900K .......... .......... .......... .......... .......... 19% 25.0M 6s 2024-03-29T04:10:05.654 INFO:teuthology.orchestra.run.smithi156.stderr: 32950K .......... .......... .......... .......... .......... 19% 33.3M 6s 2024-03-29T04:10:05.656 INFO:teuthology.orchestra.run.smithi156.stderr: 33000K .......... .......... .......... .......... .......... 19% 33.5M 6s 2024-03-29T04:10:05.657 INFO:teuthology.orchestra.run.smithi156.stderr: 33050K .......... .......... .......... .......... .......... 19% 31.1M 6s 2024-03-29T04:10:05.659 INFO:teuthology.orchestra.run.smithi156.stderr: 33100K .......... .......... .......... .......... .......... 19% 30.3M 6s 2024-03-29T04:10:05.660 INFO:teuthology.orchestra.run.smithi156.stderr: 33150K .......... .......... .......... .......... .......... 19% 37.2M 6s 2024-03-29T04:10:05.662 INFO:teuthology.orchestra.run.smithi156.stderr: 33200K .......... .......... .......... .......... .......... 19% 37.1M 6s 2024-03-29T04:10:05.663 INFO:teuthology.orchestra.run.smithi156.stderr: 33250K .......... .......... .......... .......... .......... 19% 34.3M 6s 2024-03-29T04:10:05.664 INFO:teuthology.orchestra.run.smithi156.stderr: 33300K .......... .......... .......... .......... .......... 19% 34.3M 6s 2024-03-29T04:10:05.666 INFO:teuthology.orchestra.run.smithi156.stderr: 33350K .......... .......... .......... .......... .......... 19% 38.8M 6s 2024-03-29T04:10:05.667 INFO:teuthology.orchestra.run.smithi156.stderr: 33400K .......... .......... .......... .......... .......... 19% 40.1M 6s 2024-03-29T04:10:05.668 INFO:teuthology.orchestra.run.smithi156.stderr: 33450K .......... .......... .......... .......... .......... 19% 40.1M 6s 2024-03-29T04:10:05.669 INFO:teuthology.orchestra.run.smithi156.stderr: 33500K .......... .......... .......... .......... .......... 19% 33.5M 6s 2024-03-29T04:10:05.671 INFO:teuthology.orchestra.run.smithi156.stderr: 33550K .......... .......... .......... .......... .......... 19% 38.3M 6s 2024-03-29T04:10:05.675 INFO:teuthology.orchestra.run.smithi156.stderr: 33600K .......... .......... .......... .......... .......... 19% 39.4M 6s 2024-03-29T04:10:05.675 INFO:teuthology.orchestra.run.smithi156.stderr: 33650K .......... .......... .......... .......... .......... 19% 40.0M 6s 2024-03-29T04:10:05.675 INFO:teuthology.orchestra.run.smithi156.stderr: 33700K .......... .......... .......... .......... .......... 20% 34.4M 6s 2024-03-29T04:10:05.679 INFO:teuthology.orchestra.run.smithi156.stderr: 33750K .......... .......... .......... .......... .......... 20% 40.3M 6s 2024-03-29T04:10:05.679 INFO:teuthology.orchestra.run.smithi156.stderr: 33800K .......... .......... .......... .......... .......... 20% 39.6M 6s 2024-03-29T04:10:05.679 INFO:teuthology.orchestra.run.smithi156.stderr: 33850K .......... .......... .......... .......... .......... 20% 39.1M 6s 2024-03-29T04:10:05.681 INFO:teuthology.orchestra.run.smithi156.stderr: 33900K .......... .......... .......... .......... .......... 20% 33.2M 6s 2024-03-29T04:10:05.681 INFO:teuthology.orchestra.run.smithi156.stderr: 33950K .......... .......... .......... .......... .......... 20% 38.1M 6s 2024-03-29T04:10:05.683 INFO:teuthology.orchestra.run.smithi156.stderr: 34000K .......... .......... .......... .......... .......... 20% 37.6M 6s 2024-03-29T04:10:05.684 INFO:teuthology.orchestra.run.smithi156.stderr: 34050K .......... .......... .......... .......... .......... 20% 39.1M 6s 2024-03-29T04:10:05.685 INFO:teuthology.orchestra.run.smithi156.stderr: 34100K .......... .......... .......... .......... .......... 20% 32.8M 6s 2024-03-29T04:10:05.686 INFO:teuthology.orchestra.run.smithi156.stderr: 34150K .......... .......... .......... .......... .......... 20% 37.3M 6s 2024-03-29T04:10:05.688 INFO:teuthology.orchestra.run.smithi156.stderr: 34200K .......... .......... .......... .......... .......... 20% 38.0M 6s 2024-03-29T04:10:05.689 INFO:teuthology.orchestra.run.smithi156.stderr: 34250K .......... .......... .......... .......... .......... 20% 37.1M 6s 2024-03-29T04:10:05.691 INFO:teuthology.orchestra.run.smithi156.stderr: 34300K .......... .......... .......... .......... .......... 20% 28.0M 6s 2024-03-29T04:10:05.692 INFO:teuthology.orchestra.run.smithi156.stderr: 34350K .......... .......... .......... .......... .......... 20% 34.9M 6s 2024-03-29T04:10:05.693 INFO:teuthology.orchestra.run.smithi156.stderr: 34400K .......... .......... .......... .......... .......... 20% 34.2M 6s 2024-03-29T04:10:05.695 INFO:teuthology.orchestra.run.smithi156.stderr: 34450K .......... .......... .......... .......... .......... 20% 34.0M 6s 2024-03-29T04:10:05.697 INFO:teuthology.orchestra.run.smithi156.stderr: 34500K .......... .......... .......... .......... .......... 20% 29.1M 6s 2024-03-29T04:10:05.698 INFO:teuthology.orchestra.run.smithi156.stderr: 34550K .......... .......... .......... .......... .......... 20% 35.0M 6s 2024-03-29T04:10:05.699 INFO:teuthology.orchestra.run.smithi156.stderr: 34600K .......... .......... .......... .......... .......... 20% 35.0M 6s 2024-03-29T04:10:05.701 INFO:teuthology.orchestra.run.smithi156.stderr: 34650K .......... .......... .......... .......... .......... 20% 34.7M 6s 2024-03-29T04:10:05.709 INFO:teuthology.orchestra.run.smithi156.stderr: 34700K .......... .......... .......... .......... .......... 20% 30.0M 6s 2024-03-29T04:10:05.709 INFO:teuthology.orchestra.run.smithi156.stderr: 34750K .......... .......... .......... .......... .......... 20% 34.7M 6s 2024-03-29T04:10:05.709 INFO:teuthology.orchestra.run.smithi156.stderr: 34800K .......... .......... .......... .......... .......... 20% 35.0M 6s 2024-03-29T04:10:05.709 INFO:teuthology.orchestra.run.smithi156.stderr: 34850K .......... .......... .......... .......... .......... 20% 35.5M 6s 2024-03-29T04:10:05.709 INFO:teuthology.orchestra.run.smithi156.stderr: 34900K .......... .......... .......... .......... .......... 20% 30.2M 6s 2024-03-29T04:10:05.710 INFO:teuthology.orchestra.run.smithi156.stderr: 34950K .......... .......... .......... .......... .......... 20% 33.4M 6s 2024-03-29T04:10:05.711 INFO:teuthology.orchestra.run.smithi156.stderr: 35000K .......... .......... .......... .......... .......... 20% 33.5M 6s 2024-03-29T04:10:05.713 INFO:teuthology.orchestra.run.smithi156.stderr: 35050K .......... .......... .......... .......... .......... 20% 34.0M 6s 2024-03-29T04:10:05.714 INFO:teuthology.orchestra.run.smithi156.stderr: 35100K .......... .......... .......... .......... .......... 20% 28.7M 6s 2024-03-29T04:10:05.716 INFO:teuthology.orchestra.run.smithi156.stderr: 35150K .......... .......... .......... .......... .......... 20% 33.2M 6s 2024-03-29T04:10:05.717 INFO:teuthology.orchestra.run.smithi156.stderr: 35200K .......... .......... .......... .......... .......... 20% 32.2M 6s 2024-03-29T04:10:05.719 INFO:teuthology.orchestra.run.smithi156.stderr: 35250K .......... .......... .......... .......... .......... 20% 31.6M 6s 2024-03-29T04:10:05.721 INFO:teuthology.orchestra.run.smithi156.stderr: 35300K .......... .......... .......... .......... .......... 20% 26.1M 6s 2024-03-29T04:10:05.722 INFO:teuthology.orchestra.run.smithi156.stderr: 35350K .......... .......... .......... .......... .......... 21% 30.7M 6s 2024-03-29T04:10:05.724 INFO:teuthology.orchestra.run.smithi156.stderr: 35400K .......... .......... .......... .......... .......... 21% 30.8M 6s 2024-03-29T04:10:05.725 INFO:teuthology.orchestra.run.smithi156.stderr: 35450K .......... .......... .......... .......... .......... 21% 32.9M 6s 2024-03-29T04:10:05.727 INFO:teuthology.orchestra.run.smithi156.stderr: 35500K .......... .......... .......... .......... .......... 21% 21.7M 6s 2024-03-29T04:10:05.730 INFO:teuthology.orchestra.run.smithi156.stderr: 35550K .......... .......... .......... .......... .......... 21% 24.6M 6s 2024-03-29T04:10:05.732 INFO:teuthology.orchestra.run.smithi156.stderr: 35600K .......... .......... .......... .......... .......... 21% 24.3M 6s 2024-03-29T04:10:05.734 INFO:teuthology.orchestra.run.smithi156.stderr: 35650K .......... .......... .......... .......... .......... 21% 24.0M 6s 2024-03-29T04:10:05.736 INFO:teuthology.orchestra.run.smithi156.stderr: 35700K .......... .......... .......... .......... .......... 21% 19.9M 6s 2024-03-29T04:10:05.738 INFO:teuthology.orchestra.run.smithi156.stderr: 35750K .......... .......... .......... .......... .......... 21% 23.9M 6s 2024-03-29T04:10:05.740 INFO:teuthology.orchestra.run.smithi156.stderr: 35800K .......... .......... .......... .......... .......... 21% 23.6M 6s 2024-03-29T04:10:05.743 INFO:teuthology.orchestra.run.smithi156.stderr: 35850K .......... .......... .......... .......... .......... 21% 23.7M 6s 2024-03-29T04:10:05.745 INFO:teuthology.orchestra.run.smithi156.stderr: 35900K .......... .......... .......... .......... .......... 21% 20.2M 6s 2024-03-29T04:10:05.747 INFO:teuthology.orchestra.run.smithi156.stderr: 35950K .......... .......... .......... .......... .......... 21% 23.2M 6s 2024-03-29T04:10:05.749 INFO:teuthology.orchestra.run.smithi156.stderr: 36000K .......... .......... .......... .......... .......... 21% 24.0M 6s 2024-03-29T04:10:05.753 INFO:teuthology.orchestra.run.smithi156.stderr: 36050K .......... .......... .......... .......... .......... 21% 23.9M 6s 2024-03-29T04:10:05.753 INFO:teuthology.orchestra.run.smithi156.stderr: 36100K .......... .......... .......... .......... .......... 21% 20.1M 6s 2024-03-29T04:10:05.755 INFO:teuthology.orchestra.run.smithi156.stderr: 36150K .......... .......... .......... .......... .......... 21% 23.3M 6s 2024-03-29T04:10:05.758 INFO:teuthology.orchestra.run.smithi156.stderr: 36200K .......... .......... .......... .......... .......... 21% 23.4M 6s 2024-03-29T04:10:05.760 INFO:teuthology.orchestra.run.smithi156.stderr: 36250K .......... .......... .......... .......... .......... 21% 23.0M 6s 2024-03-29T04:10:05.762 INFO:teuthology.orchestra.run.smithi156.stderr: 36300K .......... .......... .......... .......... .......... 21% 20.1M 6s 2024-03-29T04:10:05.764 INFO:teuthology.orchestra.run.smithi156.stderr: 36350K .......... .......... .......... .......... .......... 21% 23.0M 6s 2024-03-29T04:10:05.767 INFO:teuthology.orchestra.run.smithi156.stderr: 36400K .......... .......... .......... .......... .......... 21% 20.3M 6s 2024-03-29T04:10:05.769 INFO:teuthology.orchestra.run.smithi156.stderr: 36450K .......... .......... .......... .......... .......... 21% 19.7M 6s 2024-03-29T04:10:05.771 INFO:teuthology.orchestra.run.smithi156.stderr: 36500K .......... .......... .......... .......... .......... 21% 20.7M 6s 2024-03-29T04:10:05.773 INFO:teuthology.orchestra.run.smithi156.stderr: 36550K .......... .......... .......... .......... .......... 21% 28.8M 6s 2024-03-29T04:10:05.775 INFO:teuthology.orchestra.run.smithi156.stderr: 36600K .......... .......... .......... .......... .......... 21% 28.4M 6s 2024-03-29T04:10:05.777 INFO:teuthology.orchestra.run.smithi156.stderr: 36650K .......... .......... .......... .......... .......... 21% 28.5M 6s 2024-03-29T04:10:05.778 INFO:teuthology.orchestra.run.smithi156.stderr: 36700K .......... .......... .......... .......... .......... 21% 24.4M 6s 2024-03-29T04:10:05.780 INFO:teuthology.orchestra.run.smithi156.stderr: 36750K .......... .......... .......... .......... .......... 21% 29.7M 6s 2024-03-29T04:10:05.782 INFO:teuthology.orchestra.run.smithi156.stderr: 36800K .......... .......... .......... .......... .......... 21% 26.5M 6s 2024-03-29T04:10:05.785 INFO:teuthology.orchestra.run.smithi156.stderr: 36850K .......... .......... .......... .......... .......... 21% 26.4M 6s 2024-03-29T04:10:05.786 INFO:teuthology.orchestra.run.smithi156.stderr: 36900K .......... .......... .......... .......... .......... 21% 33.1M 6s 2024-03-29T04:10:05.787 INFO:teuthology.orchestra.run.smithi156.stderr: 36950K .......... .......... .......... .......... .......... 21% 29.9M 6s 2024-03-29T04:10:05.789 INFO:teuthology.orchestra.run.smithi156.stderr: 37000K .......... .......... .......... .......... .......... 21% 28.8M 6s 2024-03-29T04:10:05.790 INFO:teuthology.orchestra.run.smithi156.stderr: 37050K .......... .......... .......... .......... .......... 22% 30.9M 6s 2024-03-29T04:10:05.792 INFO:teuthology.orchestra.run.smithi156.stderr: 37100K .......... .......... .......... .......... .......... 22% 26.7M 6s 2024-03-29T04:10:05.794 INFO:teuthology.orchestra.run.smithi156.stderr: 37150K .......... .......... .......... .......... .......... 22% 33.8M 6s 2024-03-29T04:10:05.795 INFO:teuthology.orchestra.run.smithi156.stderr: 37200K .......... .......... .......... .......... .......... 22% 34.7M 6s 2024-03-29T04:10:05.796 INFO:teuthology.orchestra.run.smithi156.stderr: 37250K .......... .......... .......... .......... .......... 22% 34.5M 6s 2024-03-29T04:10:05.798 INFO:teuthology.orchestra.run.smithi156.stderr: 37300K .......... .......... .......... .......... .......... 22% 31.3M 6s 2024-03-29T04:10:05.799 INFO:teuthology.orchestra.run.smithi156.stderr: 37350K .......... .......... .......... .......... .......... 22% 43.5M 6s 2024-03-29T04:10:05.800 INFO:teuthology.orchestra.run.smithi156.stderr: 37400K .......... .......... .......... .......... .......... 22% 41.4M 6s 2024-03-29T04:10:05.802 INFO:teuthology.orchestra.run.smithi156.stderr: 37450K .......... .......... .......... .......... .......... 22% 41.7M 6s 2024-03-29T04:10:05.803 INFO:teuthology.orchestra.run.smithi156.stderr: 37500K .......... .......... .......... .......... .......... 22% 35.6M 6s 2024-03-29T04:10:05.804 INFO:teuthology.orchestra.run.smithi156.stderr: 37550K .......... .......... .......... .......... .......... 22% 41.2M 6s 2024-03-29T04:10:05.805 INFO:teuthology.orchestra.run.smithi156.stderr: 37600K .......... .......... .......... .......... .......... 22% 40.4M 6s 2024-03-29T04:10:05.807 INFO:teuthology.orchestra.run.smithi156.stderr: 37650K .......... .......... .......... .......... .......... 22% 35.6M 6s 2024-03-29T04:10:05.808 INFO:teuthology.orchestra.run.smithi156.stderr: 37700K .......... .......... .......... .......... .......... 22% 25.6M 6s 2024-03-29T04:10:05.810 INFO:teuthology.orchestra.run.smithi156.stderr: 37750K .......... .......... .......... .......... .......... 22% 30.7M 6s 2024-03-29T04:10:05.812 INFO:teuthology.orchestra.run.smithi156.stderr: 37800K .......... .......... .......... .......... .......... 22% 29.8M 6s 2024-03-29T04:10:05.813 INFO:teuthology.orchestra.run.smithi156.stderr: 37850K .......... .......... .......... .......... .......... 22% 30.1M 6s 2024-03-29T04:10:05.815 INFO:teuthology.orchestra.run.smithi156.stderr: 37900K .......... .......... .......... .......... .......... 22% 24.0M 6s 2024-03-29T04:10:05.817 INFO:teuthology.orchestra.run.smithi156.stderr: 37950K .......... .......... .......... .......... .......... 22% 28.9M 6s 2024-03-29T04:10:05.819 INFO:teuthology.orchestra.run.smithi156.stderr: 38000K .......... .......... .......... .......... .......... 22% 29.1M 6s 2024-03-29T04:10:05.821 INFO:teuthology.orchestra.run.smithi156.stderr: 38050K .......... .......... .......... .......... .......... 22% 27.8M 6s 2024-03-29T04:10:05.823 INFO:teuthology.orchestra.run.smithi156.stderr: 38100K .......... .......... .......... .......... .......... 22% 23.9M 6s 2024-03-29T04:10:05.824 INFO:teuthology.orchestra.run.smithi156.stderr: 38150K .......... .......... .......... .......... .......... 22% 29.2M 6s 2024-03-29T04:10:05.826 INFO:teuthology.orchestra.run.smithi156.stderr: 38200K .......... .......... .......... .......... .......... 22% 29.6M 6s 2024-03-29T04:10:05.828 INFO:teuthology.orchestra.run.smithi156.stderr: 38250K .......... .......... .......... .......... .......... 22% 30.3M 6s 2024-03-29T04:10:05.830 INFO:teuthology.orchestra.run.smithi156.stderr: 38300K .......... .......... .......... .......... .......... 22% 26.0M 6s 2024-03-29T04:10:05.831 INFO:teuthology.orchestra.run.smithi156.stderr: 38350K .......... .......... .......... .......... .......... 22% 30.0M 6s 2024-03-29T04:10:05.833 INFO:teuthology.orchestra.run.smithi156.stderr: 38400K .......... .......... .......... .......... .......... 22% 28.7M 6s 2024-03-29T04:10:05.834 INFO:teuthology.orchestra.run.smithi156.stderr: 38450K .......... .......... .......... .......... .......... 22% 30.0M 6s 2024-03-29T04:10:05.836 INFO:teuthology.orchestra.run.smithi156.stderr: 38500K .......... .......... .......... .......... .......... 22% 25.3M 6s 2024-03-29T04:10:05.838 INFO:teuthology.orchestra.run.smithi156.stderr: 38550K .......... .......... .......... .......... .......... 22% 28.9M 6s 2024-03-29T04:10:05.840 INFO:teuthology.orchestra.run.smithi156.stderr: 38600K .......... .......... .......... .......... .......... 22% 30.1M 6s 2024-03-29T04:10:05.842 INFO:teuthology.orchestra.run.smithi156.stderr: 38650K .......... .......... .......... .......... .......... 22% 26.5M 6s 2024-03-29T04:10:05.843 INFO:teuthology.orchestra.run.smithi156.stderr: 38700K .......... .......... .......... .......... .......... 22% 24.3M 6s 2024-03-29T04:10:05.845 INFO:teuthology.orchestra.run.smithi156.stderr: 38750K .......... .......... .......... .......... .......... 23% 28.9M 6s 2024-03-29T04:10:05.847 INFO:teuthology.orchestra.run.smithi156.stderr: 38800K .......... .......... .......... .......... .......... 23% 28.2M 6s 2024-03-29T04:10:05.849 INFO:teuthology.orchestra.run.smithi156.stderr: 38850K .......... .......... .......... .......... .......... 23% 28.6M 6s 2024-03-29T04:10:05.850 INFO:teuthology.orchestra.run.smithi156.stderr: 38900K .......... .......... .......... .......... .......... 23% 25.0M 6s 2024-03-29T04:10:05.852 INFO:teuthology.orchestra.run.smithi156.stderr: 38950K .......... .......... .......... .......... .......... 23% 28.2M 6s 2024-03-29T04:10:05.854 INFO:teuthology.orchestra.run.smithi156.stderr: 39000K .......... .......... .......... .......... .......... 23% 27.5M 6s 2024-03-29T04:10:05.856 INFO:teuthology.orchestra.run.smithi156.stderr: 39050K .......... .......... .......... .......... .......... 23% 27.4M 6s 2024-03-29T04:10:05.858 INFO:teuthology.orchestra.run.smithi156.stderr: 39100K .......... .......... .......... .......... .......... 23% 26.1M 6s 2024-03-29T04:10:05.859 INFO:teuthology.orchestra.run.smithi156.stderr: 39150K .......... .......... .......... .......... .......... 23% 31.0M 5s 2024-03-29T04:10:05.861 INFO:teuthology.orchestra.run.smithi156.stderr: 39200K .......... .......... .......... .......... .......... 23% 33.9M 5s 2024-03-29T04:10:05.862 INFO:teuthology.orchestra.run.smithi156.stderr: 39250K .......... .......... .......... .......... .......... 23% 32.0M 5s 2024-03-29T04:10:05.864 INFO:teuthology.orchestra.run.smithi156.stderr: 39300K .......... .......... .......... .......... .......... 23% 24.6M 5s 2024-03-29T04:10:05.866 INFO:teuthology.orchestra.run.smithi156.stderr: 39350K .......... .......... .......... .......... .......... 23% 26.7M 5s 2024-03-29T04:10:05.868 INFO:teuthology.orchestra.run.smithi156.stderr: 39400K .......... .......... .......... .......... .......... 23% 27.2M 5s 2024-03-29T04:10:05.870 INFO:teuthology.orchestra.run.smithi156.stderr: 39450K .......... .......... .......... .......... .......... 23% 27.8M 5s 2024-03-29T04:10:05.872 INFO:teuthology.orchestra.run.smithi156.stderr: 39500K .......... .......... .......... .......... .......... 23% 23.4M 5s 2024-03-29T04:10:05.873 INFO:teuthology.orchestra.run.smithi156.stderr: 39550K .......... .......... .......... .......... .......... 23% 26.8M 5s 2024-03-29T04:10:05.875 INFO:teuthology.orchestra.run.smithi156.stderr: 39600K .......... .......... .......... .......... .......... 23% 28.6M 5s 2024-03-29T04:10:05.877 INFO:teuthology.orchestra.run.smithi156.stderr: 39650K .......... .......... .......... .......... .......... 23% 28.0M 5s 2024-03-29T04:10:05.879 INFO:teuthology.orchestra.run.smithi156.stderr: 39700K .......... .......... .......... .......... .......... 23% 23.5M 5s 2024-03-29T04:10:05.881 INFO:teuthology.orchestra.run.smithi156.stderr: 39750K .......... .......... .......... .......... .......... 23% 27.6M 5s 2024-03-29T04:10:05.883 INFO:teuthology.orchestra.run.smithi156.stderr: 39800K .......... .......... .......... .......... .......... 23% 28.0M 5s 2024-03-29T04:10:05.884 INFO:teuthology.orchestra.run.smithi156.stderr: 39850K .......... .......... .......... .......... .......... 23% 27.4M 5s 2024-03-29T04:10:05.886 INFO:teuthology.orchestra.run.smithi156.stderr: 39900K .......... .......... .......... .......... .......... 23% 23.6M 5s 2024-03-29T04:10:05.888 INFO:teuthology.orchestra.run.smithi156.stderr: 39950K .......... .......... .......... .......... .......... 23% 27.3M 5s 2024-03-29T04:10:05.890 INFO:teuthology.orchestra.run.smithi156.stderr: 40000K .......... .......... .......... .......... .......... 23% 28.3M 5s 2024-03-29T04:10:05.892 INFO:teuthology.orchestra.run.smithi156.stderr: 40050K .......... .......... .......... .......... .......... 23% 28.5M 5s 2024-03-29T04:10:05.894 INFO:teuthology.orchestra.run.smithi156.stderr: 40100K .......... .......... .......... .......... .......... 23% 24.0M 5s 2024-03-29T04:10:05.895 INFO:teuthology.orchestra.run.smithi156.stderr: 40150K .......... .......... .......... .......... .......... 23% 28.0M 5s 2024-03-29T04:10:05.897 INFO:teuthology.orchestra.run.smithi156.stderr: 40200K .......... .......... .......... .......... .......... 23% 26.8M 5s 2024-03-29T04:10:05.899 INFO:teuthology.orchestra.run.smithi156.stderr: 40250K .......... .......... .......... .......... .......... 23% 27.3M 5s 2024-03-29T04:10:05.901 INFO:teuthology.orchestra.run.smithi156.stderr: 40300K .......... .......... .......... .......... .......... 23% 22.9M 5s 2024-03-29T04:10:05.903 INFO:teuthology.orchestra.run.smithi156.stderr: 40350K .......... .......... .......... .......... .......... 23% 26.9M 5s 2024-03-29T04:10:05.905 INFO:teuthology.orchestra.run.smithi156.stderr: 40400K .......... .......... .......... .......... .......... 23% 26.9M 5s 2024-03-29T04:10:05.912 INFO:teuthology.orchestra.run.smithi156.stderr: 40450K .......... .......... .......... .......... .......... 24% 26.9M 5s 2024-03-29T04:10:05.912 INFO:teuthology.orchestra.run.smithi156.stderr: 40500K .......... .......... .......... .......... .......... 24% 23.3M 5s 2024-03-29T04:10:05.912 INFO:teuthology.orchestra.run.smithi156.stderr: 40550K .......... .......... .......... .......... .......... 24% 27.3M 5s 2024-03-29T04:10:05.913 INFO:teuthology.orchestra.run.smithi156.stderr: 40600K .......... .......... .......... .......... .......... 24% 26.7M 5s 2024-03-29T04:10:05.914 INFO:teuthology.orchestra.run.smithi156.stderr: 40650K .......... .......... .......... .......... .......... 24% 26.6M 5s 2024-03-29T04:10:05.916 INFO:teuthology.orchestra.run.smithi156.stderr: 40700K .......... .......... .......... .......... .......... 24% 22.3M 5s 2024-03-29T04:10:05.918 INFO:teuthology.orchestra.run.smithi156.stderr: 40750K .......... .......... .......... .......... .......... 24% 26.1M 5s 2024-03-29T04:10:05.920 INFO:teuthology.orchestra.run.smithi156.stderr: 40800K .......... .......... .......... .......... .......... 24% 26.0M 5s 2024-03-29T04:10:05.922 INFO:teuthology.orchestra.run.smithi156.stderr: 40850K .......... .......... .......... .......... .......... 24% 24.8M 5s 2024-03-29T04:10:05.924 INFO:teuthology.orchestra.run.smithi156.stderr: 40900K .......... .......... .......... .......... .......... 24% 22.4M 5s 2024-03-29T04:10:05.926 INFO:teuthology.orchestra.run.smithi156.stderr: 40950K .......... .......... .......... .......... .......... 24% 28.5M 5s 2024-03-29T04:10:05.929 INFO:teuthology.orchestra.run.smithi156.stderr: 41000K .......... .......... .......... .......... .......... 24% 33.0M 5s 2024-03-29T04:10:05.930 INFO:teuthology.orchestra.run.smithi156.stderr: 41050K .......... .......... .......... .......... .......... 24% 26.5M 5s 2024-03-29T04:10:05.931 INFO:teuthology.orchestra.run.smithi156.stderr: 41100K .......... .......... .......... .......... .......... 24% 24.8M 5s 2024-03-29T04:10:05.933 INFO:teuthology.orchestra.run.smithi156.stderr: 41150K .......... .......... .......... .......... .......... 24% 31.6M 5s 2024-03-29T04:10:05.934 INFO:teuthology.orchestra.run.smithi156.stderr: 41200K .......... .......... .......... .......... .......... 24% 33.0M 5s 2024-03-29T04:10:05.936 INFO:teuthology.orchestra.run.smithi156.stderr: 41250K .......... .......... .......... .......... .......... 24% 34.5M 5s 2024-03-29T04:10:05.937 INFO:teuthology.orchestra.run.smithi156.stderr: 41300K .......... .......... .......... .......... .......... 24% 34.0M 5s 2024-03-29T04:10:05.938 INFO:teuthology.orchestra.run.smithi156.stderr: 41350K .......... .......... .......... .......... .......... 24% 40.4M 5s 2024-03-29T04:10:05.939 INFO:teuthology.orchestra.run.smithi156.stderr: 41400K .......... .......... .......... .......... .......... 24% 40.1M 5s 2024-03-29T04:10:05.941 INFO:teuthology.orchestra.run.smithi156.stderr: 41450K .......... .......... .......... .......... .......... 24% 37.0M 5s 2024-03-29T04:10:05.942 INFO:teuthology.orchestra.run.smithi156.stderr: 41500K .......... .......... .......... .......... .......... 24% 28.5M 5s 2024-03-29T04:10:05.944 INFO:teuthology.orchestra.run.smithi156.stderr: 41550K .......... .......... .......... .......... .......... 24% 40.0M 5s 2024-03-29T04:10:05.945 INFO:teuthology.orchestra.run.smithi156.stderr: 41600K .......... .......... .......... .......... .......... 24% 39.0M 5s 2024-03-29T04:10:05.946 INFO:teuthology.orchestra.run.smithi156.stderr: 41650K .......... .......... .......... .......... .......... 24% 39.1M 5s 2024-03-29T04:10:05.948 INFO:teuthology.orchestra.run.smithi156.stderr: 41700K .......... .......... .......... .......... .......... 24% 33.9M 5s 2024-03-29T04:10:05.949 INFO:teuthology.orchestra.run.smithi156.stderr: 41750K .......... .......... .......... .......... .......... 24% 37.8M 5s 2024-03-29T04:10:05.950 INFO:teuthology.orchestra.run.smithi156.stderr: 41800K .......... .......... .......... .......... .......... 24% 38.3M 5s 2024-03-29T04:10:05.951 INFO:teuthology.orchestra.run.smithi156.stderr: 41850K .......... .......... .......... .......... .......... 24% 38.2M 5s 2024-03-29T04:10:05.953 INFO:teuthology.orchestra.run.smithi156.stderr: 41900K .......... .......... .......... .......... .......... 24% 33.1M 5s 2024-03-29T04:10:05.954 INFO:teuthology.orchestra.run.smithi156.stderr: 41950K .......... .......... .......... .......... .......... 24% 37.1M 5s 2024-03-29T04:10:05.956 INFO:teuthology.orchestra.run.smithi156.stderr: 42000K .......... .......... .......... .......... .......... 24% 38.0M 5s 2024-03-29T04:10:05.957 INFO:teuthology.orchestra.run.smithi156.stderr: 42050K .......... .......... .......... .......... .......... 24% 37.5M 5s 2024-03-29T04:10:05.958 INFO:teuthology.orchestra.run.smithi156.stderr: 42100K .......... .......... .......... .......... .......... 25% 32.0M 5s 2024-03-29T04:10:05.960 INFO:teuthology.orchestra.run.smithi156.stderr: 42150K .......... .......... .......... .......... .......... 25% 36.5M 5s 2024-03-29T04:10:05.961 INFO:teuthology.orchestra.run.smithi156.stderr: 42200K .......... .......... .......... .......... .......... 25% 36.7M 5s 2024-03-29T04:10:05.962 INFO:teuthology.orchestra.run.smithi156.stderr: 42250K .......... .......... .......... .......... .......... 25% 36.7M 5s 2024-03-29T04:10:05.964 INFO:teuthology.orchestra.run.smithi156.stderr: 42300K .......... .......... .......... .......... .......... 25% 31.2M 5s 2024-03-29T04:10:05.965 INFO:teuthology.orchestra.run.smithi156.stderr: 42350K .......... .......... .......... .......... .......... 25% 36.3M 5s 2024-03-29T04:10:05.967 INFO:teuthology.orchestra.run.smithi156.stderr: 42400K .......... .......... .......... .......... .......... 25% 36.4M 5s 2024-03-29T04:10:05.968 INFO:teuthology.orchestra.run.smithi156.stderr: 42450K .......... .......... .......... .......... .......... 25% 36.3M 5s 2024-03-29T04:10:05.970 INFO:teuthology.orchestra.run.smithi156.stderr: 42500K .......... .......... .......... .......... .......... 25% 31.4M 5s 2024-03-29T04:10:05.971 INFO:teuthology.orchestra.run.smithi156.stderr: 42550K .......... .......... .......... .......... .......... 25% 35.0M 5s 2024-03-29T04:10:05.972 INFO:teuthology.orchestra.run.smithi156.stderr: 42600K .......... .......... .......... .......... .......... 25% 33.2M 5s 2024-03-29T04:10:05.974 INFO:teuthology.orchestra.run.smithi156.stderr: 42650K .......... .......... .......... .......... .......... 25% 34.6M 5s 2024-03-29T04:10:05.975 INFO:teuthology.orchestra.run.smithi156.stderr: 42700K .......... .......... .......... .......... .......... 25% 29.9M 5s 2024-03-29T04:10:05.977 INFO:teuthology.orchestra.run.smithi156.stderr: 42750K .......... .......... .......... .......... .......... 25% 35.3M 5s 2024-03-29T04:10:05.978 INFO:teuthology.orchestra.run.smithi156.stderr: 42800K .......... .......... .......... .......... .......... 25% 35.9M 5s 2024-03-29T04:10:05.980 INFO:teuthology.orchestra.run.smithi156.stderr: 42850K .......... .......... .......... .......... .......... 25% 35.9M 5s 2024-03-29T04:10:05.982 INFO:teuthology.orchestra.run.smithi156.stderr: 42900K .......... .......... .......... .......... .......... 25% 30.8M 5s 2024-03-29T04:10:05.983 INFO:teuthology.orchestra.run.smithi156.stderr: 42950K .......... .......... .......... .......... .......... 25% 36.0M 5s 2024-03-29T04:10:05.984 INFO:teuthology.orchestra.run.smithi156.stderr: 43000K .......... .......... .......... .......... .......... 25% 35.3M 5s 2024-03-29T04:10:05.985 INFO:teuthology.orchestra.run.smithi156.stderr: 43050K .......... .......... .......... .......... .......... 25% 34.7M 5s 2024-03-29T04:10:05.987 INFO:teuthology.orchestra.run.smithi156.stderr: 43100K .......... .......... .......... .......... .......... 25% 30.4M 5s 2024-03-29T04:10:05.989 INFO:teuthology.orchestra.run.smithi156.stderr: 43150K .......... .......... .......... .......... .......... 25% 35.0M 5s 2024-03-29T04:10:05.990 INFO:teuthology.orchestra.run.smithi156.stderr: 43200K .......... .......... .......... .......... .......... 25% 35.1M 5s 2024-03-29T04:10:05.991 INFO:teuthology.orchestra.run.smithi156.stderr: 43250K .......... .......... .......... .......... .......... 25% 31.3M 5s 2024-03-29T04:10:05.993 INFO:teuthology.orchestra.run.smithi156.stderr: 43300K .......... .......... .......... .......... .......... 25% 28.1M 5s 2024-03-29T04:10:05.994 INFO:teuthology.orchestra.run.smithi156.stderr: 43350K .......... .......... .......... .......... .......... 25% 33.2M 5s 2024-03-29T04:10:05.996 INFO:teuthology.orchestra.run.smithi156.stderr: 43400K .......... .......... .......... .......... .......... 25% 31.9M 5s 2024-03-29T04:10:05.998 INFO:teuthology.orchestra.run.smithi156.stderr: 43450K .......... .......... .......... .......... .......... 25% 31.7M 5s 2024-03-29T04:10:05.999 INFO:teuthology.orchestra.run.smithi156.stderr: 43500K .......... .......... .......... .......... .......... 25% 27.8M 5s 2024-03-29T04:10:06.001 INFO:teuthology.orchestra.run.smithi156.stderr: 43550K .......... .......... .......... .......... .......... 25% 31.6M 5s 2024-03-29T04:10:06.002 INFO:teuthology.orchestra.run.smithi156.stderr: 43600K .......... .......... .......... .......... .......... 25% 31.8M 5s 2024-03-29T04:10:06.004 INFO:teuthology.orchestra.run.smithi156.stderr: 43650K .......... .......... .......... .......... .......... 25% 31.6M 5s 2024-03-29T04:10:06.006 INFO:teuthology.orchestra.run.smithi156.stderr: 43700K .......... .......... .......... .......... .......... 25% 26.9M 5s 2024-03-29T04:10:06.007 INFO:teuthology.orchestra.run.smithi156.stderr: 43750K .......... .......... .......... .......... .......... 25% 31.8M 5s 2024-03-29T04:10:06.009 INFO:teuthology.orchestra.run.smithi156.stderr: 43800K .......... .......... .......... .......... .......... 26% 31.8M 5s 2024-03-29T04:10:06.010 INFO:teuthology.orchestra.run.smithi156.stderr: 43850K .......... .......... .......... .......... .......... 26% 32.1M 5s 2024-03-29T04:10:06.012 INFO:teuthology.orchestra.run.smithi156.stderr: 43900K .......... .......... .......... .......... .......... 26% 27.3M 5s 2024-03-29T04:10:06.014 INFO:teuthology.orchestra.run.smithi156.stderr: 43950K .......... .......... .......... .......... .......... 26% 31.6M 5s 2024-03-29T04:10:06.015 INFO:teuthology.orchestra.run.smithi156.stderr: 44000K .......... .......... .......... .......... .......... 26% 32.1M 5s 2024-03-29T04:10:06.017 INFO:teuthology.orchestra.run.smithi156.stderr: 44050K .......... .......... .......... .......... .......... 26% 31.7M 5s 2024-03-29T04:10:06.018 INFO:teuthology.orchestra.run.smithi156.stderr: 44100K .......... .......... .......... .......... .......... 26% 26.6M 5s 2024-03-29T04:10:06.020 INFO:teuthology.orchestra.run.smithi156.stderr: 44150K .......... .......... .......... .......... .......... 26% 29.8M 5s 2024-03-29T04:10:06.022 INFO:teuthology.orchestra.run.smithi156.stderr: 44200K .......... .......... .......... .......... .......... 26% 30.1M 5s 2024-03-29T04:10:06.024 INFO:teuthology.orchestra.run.smithi156.stderr: 44250K .......... .......... .......... .......... .......... 26% 29.7M 5s 2024-03-29T04:10:06.025 INFO:teuthology.orchestra.run.smithi156.stderr: 44300K .......... .......... .......... .......... .......... 26% 25.4M 5s 2024-03-29T04:10:06.027 INFO:teuthology.orchestra.run.smithi156.stderr: 44350K .......... .......... .......... .......... .......... 26% 30.1M 5s 2024-03-29T04:10:06.029 INFO:teuthology.orchestra.run.smithi156.stderr: 44400K .......... .......... .......... .......... .......... 26% 30.6M 5s 2024-03-29T04:10:06.030 INFO:teuthology.orchestra.run.smithi156.stderr: 44450K .......... .......... .......... .......... .......... 26% 29.7M 5s 2024-03-29T04:10:06.032 INFO:teuthology.orchestra.run.smithi156.stderr: 44500K .......... .......... .......... .......... .......... 26% 24.6M 5s 2024-03-29T04:10:06.034 INFO:teuthology.orchestra.run.smithi156.stderr: 44550K .......... .......... .......... .......... .......... 26% 30.0M 5s 2024-03-29T04:10:06.035 INFO:teuthology.orchestra.run.smithi156.stderr: 44600K .......... .......... .......... .......... .......... 26% 30.3M 5s 2024-03-29T04:10:06.037 INFO:teuthology.orchestra.run.smithi156.stderr: 44650K .......... .......... .......... .......... .......... 26% 30.4M 5s 2024-03-29T04:10:06.039 INFO:teuthology.orchestra.run.smithi156.stderr: 44700K .......... .......... .......... .......... .......... 26% 23.8M 5s 2024-03-29T04:10:06.041 INFO:teuthology.orchestra.run.smithi156.stderr: 44750K .......... .......... .......... .......... .......... 26% 26.4M 5s 2024-03-29T04:10:06.043 INFO:teuthology.orchestra.run.smithi156.stderr: 44800K .......... .......... .......... .......... .......... 26% 23.4M 5s 2024-03-29T04:10:06.045 INFO:teuthology.orchestra.run.smithi156.stderr: 44850K .......... .......... .......... .......... .......... 26% 23.5M 5s 2024-03-29T04:10:06.048 INFO:teuthology.orchestra.run.smithi156.stderr: 44900K .......... .......... .......... .......... .......... 26% 20.3M 5s 2024-03-29T04:10:06.050 INFO:teuthology.orchestra.run.smithi156.stderr: 44950K .......... .......... .......... .......... .......... 26% 23.5M 5s 2024-03-29T04:10:06.052 INFO:teuthology.orchestra.run.smithi156.stderr: 45000K .......... .......... .......... .......... .......... 26% 24.2M 5s 2024-03-29T04:10:06.054 INFO:teuthology.orchestra.run.smithi156.stderr: 45050K .......... .......... .......... .......... .......... 26% 24.3M 5s 2024-03-29T04:10:06.055 INFO:teuthology.orchestra.run.smithi156.stderr: 45100K .......... .......... .......... .......... .......... 26% 29.5M 5s 2024-03-29T04:10:06.057 INFO:teuthology.orchestra.run.smithi156.stderr: 45150K .......... .......... .......... .......... .......... 26% 31.1M 5s 2024-03-29T04:10:06.058 INFO:teuthology.orchestra.run.smithi156.stderr: 45200K .......... .......... .......... .......... .......... 26% 32.6M 5s 2024-03-29T04:10:06.060 INFO:teuthology.orchestra.run.smithi156.stderr: 45250K .......... .......... .......... .......... .......... 26% 31.8M 5s 2024-03-29T04:10:06.062 INFO:teuthology.orchestra.run.smithi156.stderr: 45300K .......... .......... .......... .......... .......... 26% 30.2M 5s 2024-03-29T04:10:06.063 INFO:teuthology.orchestra.run.smithi156.stderr: 45350K .......... .......... .......... .......... .......... 26% 31.9M 5s 2024-03-29T04:10:06.064 INFO:teuthology.orchestra.run.smithi156.stderr: 45400K .......... .......... .......... .......... .......... 26% 33.4M 5s 2024-03-29T04:10:06.066 INFO:teuthology.orchestra.run.smithi156.stderr: 45450K .......... .......... .......... .......... .......... 26% 32.2M 5s 2024-03-29T04:10:06.068 INFO:teuthology.orchestra.run.smithi156.stderr: 45500K .......... .......... .......... .......... .......... 27% 25.5M 5s 2024-03-29T04:10:06.069 INFO:teuthology.orchestra.run.smithi156.stderr: 45550K .......... .......... .......... .......... .......... 27% 31.9M 5s 2024-03-29T04:10:06.071 INFO:teuthology.orchestra.run.smithi156.stderr: 45600K .......... .......... .......... .......... .......... 27% 32.5M 5s 2024-03-29T04:10:06.073 INFO:teuthology.orchestra.run.smithi156.stderr: 45650K .......... .......... .......... .......... .......... 27% 33.3M 5s 2024-03-29T04:10:06.074 INFO:teuthology.orchestra.run.smithi156.stderr: 45700K .......... .......... .......... .......... .......... 27% 28.8M 5s 2024-03-29T04:10:06.076 INFO:teuthology.orchestra.run.smithi156.stderr: 45750K .......... .......... .......... .......... .......... 27% 31.5M 5s 2024-03-29T04:10:06.077 INFO:teuthology.orchestra.run.smithi156.stderr: 45800K .......... .......... .......... .......... .......... 27% 29.7M 5s 2024-03-29T04:10:06.080 INFO:teuthology.orchestra.run.smithi156.stderr: 45850K .......... .......... .......... .......... .......... 27% 29.5M 5s 2024-03-29T04:10:06.081 INFO:teuthology.orchestra.run.smithi156.stderr: 45900K .......... .......... .......... .......... .......... 27% 26.2M 5s 2024-03-29T04:10:06.082 INFO:teuthology.orchestra.run.smithi156.stderr: 45950K .......... .......... .......... .......... .......... 27% 30.8M 5s 2024-03-29T04:10:06.084 INFO:teuthology.orchestra.run.smithi156.stderr: 46000K .......... .......... .......... .......... .......... 27% 30.7M 5s 2024-03-29T04:10:06.086 INFO:teuthology.orchestra.run.smithi156.stderr: 46050K .......... .......... .......... .......... .......... 27% 30.9M 5s 2024-03-29T04:10:06.087 INFO:teuthology.orchestra.run.smithi156.stderr: 46100K .......... .......... .......... .......... .......... 27% 26.7M 5s 2024-03-29T04:10:06.089 INFO:teuthology.orchestra.run.smithi156.stderr: 46150K .......... .......... .......... .......... .......... 27% 30.5M 5s 2024-03-29T04:10:06.091 INFO:teuthology.orchestra.run.smithi156.stderr: 46200K .......... .......... .......... .......... .......... 27% 30.2M 5s 2024-03-29T04:10:06.092 INFO:teuthology.orchestra.run.smithi156.stderr: 46250K .......... .......... .......... .......... .......... 27% 29.0M 5s 2024-03-29T04:10:06.094 INFO:teuthology.orchestra.run.smithi156.stderr: 46300K .......... .......... .......... .......... .......... 27% 25.1M 5s 2024-03-29T04:10:06.095 INFO:teuthology.orchestra.run.smithi156.stderr: 46350K .......... .......... .......... .......... .......... 27% 34.6M 5s 2024-03-29T04:10:06.097 INFO:teuthology.orchestra.run.smithi156.stderr: 46400K .......... .......... .......... .......... .......... 27% 35.3M 5s 2024-03-29T04:10:06.098 INFO:teuthology.orchestra.run.smithi156.stderr: 46450K .......... .......... .......... .......... .......... 27% 37.2M 5s 2024-03-29T04:10:06.100 INFO:teuthology.orchestra.run.smithi156.stderr: 46500K .......... .......... .......... .......... .......... 27% 31.8M 5s 2024-03-29T04:10:06.102 INFO:teuthology.orchestra.run.smithi156.stderr: 46550K .......... .......... .......... .......... .......... 27% 36.4M 5s 2024-03-29T04:10:06.103 INFO:teuthology.orchestra.run.smithi156.stderr: 46600K .......... .......... .......... .......... .......... 27% 36.4M 5s 2024-03-29T04:10:06.104 INFO:teuthology.orchestra.run.smithi156.stderr: 46650K .......... .......... .......... .......... .......... 27% 36.0M 5s 2024-03-29T04:10:06.106 INFO:teuthology.orchestra.run.smithi156.stderr: 46700K .......... .......... .......... .......... .......... 27% 30.3M 5s 2024-03-29T04:10:06.107 INFO:teuthology.orchestra.run.smithi156.stderr: 46750K .......... .......... .......... .......... .......... 27% 33.8M 5s 2024-03-29T04:10:06.108 INFO:teuthology.orchestra.run.smithi156.stderr: 46800K .......... .......... .......... .......... .......... 27% 33.9M 5s 2024-03-29T04:10:06.110 INFO:teuthology.orchestra.run.smithi156.stderr: 46850K .......... .......... .......... .......... .......... 27% 31.9M 5s 2024-03-29T04:10:06.112 INFO:teuthology.orchestra.run.smithi156.stderr: 46900K .......... .......... .......... .......... .......... 27% 28.6M 5s 2024-03-29T04:10:06.113 INFO:teuthology.orchestra.run.smithi156.stderr: 46950K .......... .......... .......... .......... .......... 27% 34.0M 5s 2024-03-29T04:10:06.114 INFO:teuthology.orchestra.run.smithi156.stderr: 47000K .......... .......... .......... .......... .......... 27% 34.0M 5s 2024-03-29T04:10:06.119 INFO:teuthology.orchestra.run.smithi156.stderr: 47050K .......... .......... .......... .......... .......... 27% 32.8M 5s 2024-03-29T04:10:06.119 INFO:teuthology.orchestra.run.smithi156.stderr: 47100K .......... .......... .......... .......... .......... 27% 25.8M 5s 2024-03-29T04:10:06.120 INFO:teuthology.orchestra.run.smithi156.stderr: 47150K .......... .......... .......... .......... .......... 28% 27.4M 5s 2024-03-29T04:10:06.121 INFO:teuthology.orchestra.run.smithi156.stderr: 47200K .......... .......... .......... .......... .......... 28% 28.7M 5s 2024-03-29T04:10:06.123 INFO:teuthology.orchestra.run.smithi156.stderr: 47250K .......... .......... .......... .......... .......... 28% 27.1M 5s 2024-03-29T04:10:06.125 INFO:teuthology.orchestra.run.smithi156.stderr: 47300K .......... .......... .......... .......... .......... 28% 24.2M 5s 2024-03-29T04:10:06.127 INFO:teuthology.orchestra.run.smithi156.stderr: 47350K .......... .......... .......... .......... .......... 28% 28.0M 5s 2024-03-29T04:10:06.129 INFO:teuthology.orchestra.run.smithi156.stderr: 47400K .......... .......... .......... .......... .......... 28% 27.9M 5s 2024-03-29T04:10:06.131 INFO:teuthology.orchestra.run.smithi156.stderr: 47450K .......... .......... .......... .......... .......... 28% 27.9M 5s 2024-03-29T04:10:06.133 INFO:teuthology.orchestra.run.smithi156.stderr: 47500K .......... .......... .......... .......... .......... 28% 24.6M 5s 2024-03-29T04:10:06.134 INFO:teuthology.orchestra.run.smithi156.stderr: 47550K .......... .......... .......... .......... .......... 28% 28.3M 5s 2024-03-29T04:10:06.136 INFO:teuthology.orchestra.run.smithi156.stderr: 47600K .......... .......... .......... .......... .......... 28% 28.0M 5s 2024-03-29T04:10:06.138 INFO:teuthology.orchestra.run.smithi156.stderr: 47650K .......... .......... .......... .......... .......... 28% 28.6M 5s 2024-03-29T04:10:06.140 INFO:teuthology.orchestra.run.smithi156.stderr: 47700K .......... .......... .......... .......... .......... 28% 24.1M 5s 2024-03-29T04:10:06.141 INFO:teuthology.orchestra.run.smithi156.stderr: 47750K .......... .......... .......... .......... .......... 28% 27.6M 5s 2024-03-29T04:10:06.143 INFO:teuthology.orchestra.run.smithi156.stderr: 47800K .......... .......... .......... .......... .......... 28% 28.0M 5s 2024-03-29T04:10:06.145 INFO:teuthology.orchestra.run.smithi156.stderr: 47850K .......... .......... .......... .......... .......... 28% 27.5M 5s 2024-03-29T04:10:06.147 INFO:teuthology.orchestra.run.smithi156.stderr: 47900K .......... .......... .......... .......... .......... 28% 23.6M 5s 2024-03-29T04:10:06.149 INFO:teuthology.orchestra.run.smithi156.stderr: 47950K .......... .......... .......... .......... .......... 28% 27.7M 5s 2024-03-29T04:10:06.150 INFO:teuthology.orchestra.run.smithi156.stderr: 48000K .......... .......... .......... .......... .......... 28% 27.5M 5s 2024-03-29T04:10:06.153 INFO:teuthology.orchestra.run.smithi156.stderr: 48050K .......... .......... .......... .......... .......... 28% 27.6M 5s 2024-03-29T04:10:06.154 INFO:teuthology.orchestra.run.smithi156.stderr: 48100K .......... .......... .......... .......... .......... 28% 23.5M 5s 2024-03-29T04:10:06.156 INFO:teuthology.orchestra.run.smithi156.stderr: 48150K .......... .......... .......... .......... .......... 28% 27.3M 5s 2024-03-29T04:10:06.158 INFO:teuthology.orchestra.run.smithi156.stderr: 48200K .......... .......... .......... .......... .......... 28% 27.1M 5s 2024-03-29T04:10:06.160 INFO:teuthology.orchestra.run.smithi156.stderr: 48250K .......... .......... .......... .......... .......... 28% 26.7M 5s 2024-03-29T04:10:06.162 INFO:teuthology.orchestra.run.smithi156.stderr: 48300K .......... .......... .......... .......... .......... 28% 23.4M 5s 2024-03-29T04:10:06.164 INFO:teuthology.orchestra.run.smithi156.stderr: 48350K .......... .......... .......... .......... .......... 28% 26.9M 5s 2024-03-29T04:10:06.166 INFO:teuthology.orchestra.run.smithi156.stderr: 48400K .......... .......... .......... .......... .......... 28% 26.3M 5s 2024-03-29T04:10:06.168 INFO:teuthology.orchestra.run.smithi156.stderr: 48450K .......... .......... .......... .......... .......... 28% 26.2M 5s 2024-03-29T04:10:06.169 INFO:teuthology.orchestra.run.smithi156.stderr: 48500K .......... .......... .......... .......... .......... 28% 24.3M 5s 2024-03-29T04:10:06.171 INFO:teuthology.orchestra.run.smithi156.stderr: 48550K .......... .......... .......... .......... .......... 28% 31.2M 5s 2024-03-29T04:10:06.173 INFO:teuthology.orchestra.run.smithi156.stderr: 48600K .......... .......... .......... .......... .......... 28% 31.0M 5s 2024-03-29T04:10:06.174 INFO:teuthology.orchestra.run.smithi156.stderr: 48650K .......... .......... .......... .......... .......... 28% 31.4M 5s 2024-03-29T04:10:06.176 INFO:teuthology.orchestra.run.smithi156.stderr: 48700K .......... .......... .......... .......... .......... 28% 20.9M 5s 2024-03-29T04:10:06.179 INFO:teuthology.orchestra.run.smithi156.stderr: 48750K .......... .......... .......... .......... .......... 28% 23.0M 5s 2024-03-29T04:10:06.181 INFO:teuthology.orchestra.run.smithi156.stderr: 48800K .......... .......... .......... .......... .......... 28% 23.3M 5s 2024-03-29T04:10:06.183 INFO:teuthology.orchestra.run.smithi156.stderr: 48850K .......... .......... .......... .......... .......... 29% 21.8M 5s 2024-03-29T04:10:06.186 INFO:teuthology.orchestra.run.smithi156.stderr: 48900K .......... .......... .......... .......... .......... 29% 19.0M 5s 2024-03-29T04:10:06.188 INFO:teuthology.orchestra.run.smithi156.stderr: 48950K .......... .......... .......... .......... .......... 29% 21.0M 5s 2024-03-29T04:10:06.191 INFO:teuthology.orchestra.run.smithi156.stderr: 49000K .......... .......... .......... .......... .......... 29% 22.4M 5s 2024-03-29T04:10:06.193 INFO:teuthology.orchestra.run.smithi156.stderr: 49050K .......... .......... .......... .......... .......... 29% 22.2M 5s 2024-03-29T04:10:06.195 INFO:teuthology.orchestra.run.smithi156.stderr: 49100K .......... .......... .......... .......... .......... 29% 19.2M 5s 2024-03-29T04:10:06.197 INFO:teuthology.orchestra.run.smithi156.stderr: 49150K .......... .......... .......... .......... .......... 29% 27.7M 5s 2024-03-29T04:10:06.199 INFO:teuthology.orchestra.run.smithi156.stderr: 49200K .......... .......... .......... .......... .......... 29% 26.0M 5s 2024-03-29T04:10:06.200 INFO:teuthology.orchestra.run.smithi156.stderr: 49250K .......... .......... .......... .......... .......... 29% 29.9M 5s 2024-03-29T04:10:06.202 INFO:teuthology.orchestra.run.smithi156.stderr: 49300K .......... .......... .......... .......... .......... 29% 24.9M 5s 2024-03-29T04:10:06.204 INFO:teuthology.orchestra.run.smithi156.stderr: 49350K .......... .......... .......... .......... .......... 29% 30.5M 5s 2024-03-29T04:10:06.205 INFO:teuthology.orchestra.run.smithi156.stderr: 49400K .......... .......... .......... .......... .......... 29% 27.8M 5s 2024-03-29T04:10:06.207 INFO:teuthology.orchestra.run.smithi156.stderr: 49450K .......... .......... .......... .......... .......... 29% 30.1M 5s 2024-03-29T04:10:06.209 INFO:teuthology.orchestra.run.smithi156.stderr: 49500K .......... .......... .......... .......... .......... 29% 31.7M 5s 2024-03-29T04:10:06.210 INFO:teuthology.orchestra.run.smithi156.stderr: 49550K .......... .......... .......... .......... .......... 29% 36.8M 5s 2024-03-29T04:10:06.211 INFO:teuthology.orchestra.run.smithi156.stderr: 49600K .......... .......... .......... .......... .......... 29% 32.5M 5s 2024-03-29T04:10:06.213 INFO:teuthology.orchestra.run.smithi156.stderr: 49650K .......... .......... .......... .......... .......... 29% 30.3M 5s 2024-03-29T04:10:06.215 INFO:teuthology.orchestra.run.smithi156.stderr: 49700K .......... .......... .......... .......... .......... 29% 26.2M 5s 2024-03-29T04:10:06.216 INFO:teuthology.orchestra.run.smithi156.stderr: 49750K .......... .......... .......... .......... .......... 29% 33.5M 5s 2024-03-29T04:10:06.218 INFO:teuthology.orchestra.run.smithi156.stderr: 49800K .......... .......... .......... .......... .......... 29% 33.2M 5s 2024-03-29T04:10:06.219 INFO:teuthology.orchestra.run.smithi156.stderr: 49850K .......... .......... .......... .......... .......... 29% 33.1M 5s 2024-03-29T04:10:06.221 INFO:teuthology.orchestra.run.smithi156.stderr: 49900K .......... .......... .......... .......... .......... 29% 28.5M 5s 2024-03-29T04:10:06.222 INFO:teuthology.orchestra.run.smithi156.stderr: 49950K .......... .......... .......... .......... .......... 29% 31.9M 5s 2024-03-29T04:10:06.224 INFO:teuthology.orchestra.run.smithi156.stderr: 50000K .......... .......... .......... .......... .......... 29% 32.3M 5s 2024-03-29T04:10:06.226 INFO:teuthology.orchestra.run.smithi156.stderr: 50050K .......... .......... .......... .......... .......... 29% 32.9M 5s 2024-03-29T04:10:06.227 INFO:teuthology.orchestra.run.smithi156.stderr: 50100K .......... .......... .......... .......... .......... 29% 26.8M 5s 2024-03-29T04:10:06.229 INFO:teuthology.orchestra.run.smithi156.stderr: 50150K .......... .......... .......... .......... .......... 29% 31.6M 5s 2024-03-29T04:10:06.231 INFO:teuthology.orchestra.run.smithi156.stderr: 50200K .......... .......... .......... .......... .......... 29% 31.9M 5s 2024-03-29T04:10:06.232 INFO:teuthology.orchestra.run.smithi156.stderr: 50250K .......... .......... .......... .......... .......... 29% 32.3M 5s 2024-03-29T04:10:06.234 INFO:teuthology.orchestra.run.smithi156.stderr: 50300K .......... .......... .......... .......... .......... 29% 28.0M 5s 2024-03-29T04:10:06.235 INFO:teuthology.orchestra.run.smithi156.stderr: 50350K .......... .......... .......... .......... .......... 29% 32.5M 5s 2024-03-29T04:10:06.237 INFO:teuthology.orchestra.run.smithi156.stderr: 50400K .......... .......... .......... .......... .......... 29% 32.3M 5s 2024-03-29T04:10:06.238 INFO:teuthology.orchestra.run.smithi156.stderr: 50450K .......... .......... .......... .......... .......... 29% 31.9M 5s 2024-03-29T04:10:06.240 INFO:teuthology.orchestra.run.smithi156.stderr: 50500K .......... .......... .......... .......... .......... 29% 27.1M 5s 2024-03-29T04:10:06.242 INFO:teuthology.orchestra.run.smithi156.stderr: 50550K .......... .......... .......... .......... .......... 30% 28.6M 5s 2024-03-29T04:10:06.243 INFO:teuthology.orchestra.run.smithi156.stderr: 50600K .......... .......... .......... .......... .......... 30% 31.6M 5s 2024-03-29T04:10:06.245 INFO:teuthology.orchestra.run.smithi156.stderr: 50650K .......... .......... .......... .......... .......... 30% 31.4M 5s 2024-03-29T04:10:06.247 INFO:teuthology.orchestra.run.smithi156.stderr: 50700K .......... .......... .......... .......... .......... 30% 27.1M 5s 2024-03-29T04:10:06.248 INFO:teuthology.orchestra.run.smithi156.stderr: 50750K .......... .......... .......... .......... .......... 30% 31.0M 5s 2024-03-29T04:10:06.250 INFO:teuthology.orchestra.run.smithi156.stderr: 50800K .......... .......... .......... .......... .......... 30% 31.1M 5s 2024-03-29T04:10:06.252 INFO:teuthology.orchestra.run.smithi156.stderr: 50850K .......... .......... .......... .......... .......... 30% 30.9M 5s 2024-03-29T04:10:06.253 INFO:teuthology.orchestra.run.smithi156.stderr: 50900K .......... .......... .......... .......... .......... 30% 24.8M 5s 2024-03-29T04:10:06.255 INFO:teuthology.orchestra.run.smithi156.stderr: 50950K .......... .......... .......... .......... .......... 30% 32.8M 5s 2024-03-29T04:10:06.256 INFO:teuthology.orchestra.run.smithi156.stderr: 51000K .......... .......... .......... .......... .......... 30% 33.2M 5s 2024-03-29T04:10:06.258 INFO:teuthology.orchestra.run.smithi156.stderr: 51050K .......... .......... .......... .......... .......... 30% 33.6M 5s 2024-03-29T04:10:06.260 INFO:teuthology.orchestra.run.smithi156.stderr: 51100K .......... .......... .......... .......... .......... 30% 24.2M 5s 2024-03-29T04:10:06.261 INFO:teuthology.orchestra.run.smithi156.stderr: 51150K .......... .......... .......... .......... .......... 30% 27.5M 5s 2024-03-29T04:10:06.263 INFO:teuthology.orchestra.run.smithi156.stderr: 51200K .......... .......... .......... .......... .......... 30% 28.3M 5s 2024-03-29T04:10:06.266 INFO:teuthology.orchestra.run.smithi156.stderr: 51250K .......... .......... .......... .......... .......... 30% 27.9M 5s 2024-03-29T04:10:06.267 INFO:teuthology.orchestra.run.smithi156.stderr: 51300K .......... .......... .......... .......... .......... 30% 23.7M 5s 2024-03-29T04:10:06.269 INFO:teuthology.orchestra.run.smithi156.stderr: 51350K .......... .......... .......... .......... .......... 30% 26.5M 5s 2024-03-29T04:10:06.271 INFO:teuthology.orchestra.run.smithi156.stderr: 51400K .......... .......... .......... .......... .......... 30% 26.7M 5s 2024-03-29T04:10:06.273 INFO:teuthology.orchestra.run.smithi156.stderr: 51450K .......... .......... .......... .......... .......... 30% 26.2M 5s 2024-03-29T04:10:06.275 INFO:teuthology.orchestra.run.smithi156.stderr: 51500K .......... .......... .......... .......... .......... 30% 22.8M 5s 2024-03-29T04:10:06.276 INFO:teuthology.orchestra.run.smithi156.stderr: 51550K .......... .......... .......... .......... .......... 30% 27.3M 5s 2024-03-29T04:10:06.278 INFO:teuthology.orchestra.run.smithi156.stderr: 51600K .......... .......... .......... .......... .......... 30% 27.0M 5s 2024-03-29T04:10:06.280 INFO:teuthology.orchestra.run.smithi156.stderr: 51650K .......... .......... .......... .......... .......... 30% 27.5M 5s 2024-03-29T04:10:06.282 INFO:teuthology.orchestra.run.smithi156.stderr: 51700K .......... .......... .......... .......... .......... 30% 23.0M 5s 2024-03-29T04:10:06.284 INFO:teuthology.orchestra.run.smithi156.stderr: 51750K .......... .......... .......... .......... .......... 30% 26.7M 5s 2024-03-29T04:10:06.286 INFO:teuthology.orchestra.run.smithi156.stderr: 51800K .......... .......... .......... .......... .......... 30% 26.6M 5s 2024-03-29T04:10:06.288 INFO:teuthology.orchestra.run.smithi156.stderr: 51850K .......... .......... .......... .......... .......... 30% 26.7M 5s 2024-03-29T04:10:06.290 INFO:teuthology.orchestra.run.smithi156.stderr: 51900K .......... .......... .......... .......... .......... 30% 22.3M 5s 2024-03-29T04:10:06.292 INFO:teuthology.orchestra.run.smithi156.stderr: 51950K .......... .......... .......... .......... .......... 30% 26.1M 5s 2024-03-29T04:10:06.294 INFO:teuthology.orchestra.run.smithi156.stderr: 52000K .......... .......... .......... .......... .......... 30% 25.2M 5s 2024-03-29T04:10:06.296 INFO:teuthology.orchestra.run.smithi156.stderr: 52050K .......... .......... .......... .......... .......... 30% 26.7M 5s 2024-03-29T04:10:06.297 INFO:teuthology.orchestra.run.smithi156.stderr: 52100K .......... .......... .......... .......... .......... 30% 25.0M 5s 2024-03-29T04:10:06.299 INFO:teuthology.orchestra.run.smithi156.stderr: 52150K .......... .......... .......... .......... .......... 30% 29.4M 5s 2024-03-29T04:10:06.301 INFO:teuthology.orchestra.run.smithi156.stderr: 52200K .......... .......... .......... .......... .......... 30% 30.6M 5s 2024-03-29T04:10:06.302 INFO:teuthology.orchestra.run.smithi156.stderr: 52250K .......... .......... .......... .......... .......... 31% 32.1M 5s 2024-03-29T04:10:06.304 INFO:teuthology.orchestra.run.smithi156.stderr: 52300K .......... .......... .......... .......... .......... 31% 27.1M 5s 2024-03-29T04:10:06.305 INFO:teuthology.orchestra.run.smithi156.stderr: 52350K .......... .......... .......... .......... .......... 31% 31.9M 5s 2024-03-29T04:10:06.308 INFO:teuthology.orchestra.run.smithi156.stderr: 52400K .......... .......... .......... .......... .......... 31% 31.6M 5s 2024-03-29T04:10:06.310 INFO:teuthology.orchestra.run.smithi156.stderr: 52450K .......... .......... .......... .......... .......... 31% 22.0M 5s 2024-03-29T04:10:06.311 INFO:teuthology.orchestra.run.smithi156.stderr: 52500K .......... .......... .......... .......... .......... 31% 24.0M 5s 2024-03-29T04:10:06.313 INFO:teuthology.orchestra.run.smithi156.stderr: 52550K .......... .......... .......... .......... .......... 31% 27.8M 5s 2024-03-29T04:10:06.315 INFO:teuthology.orchestra.run.smithi156.stderr: 52600K .......... .......... .......... .......... .......... 31% 26.9M 5s 2024-03-29T04:10:06.317 INFO:teuthology.orchestra.run.smithi156.stderr: 52650K .......... .......... .......... .......... .......... 31% 27.3M 5s 2024-03-29T04:10:06.323 INFO:teuthology.orchestra.run.smithi156.stderr: 52700K .......... .......... .......... .......... .......... 31% 23.4M 5s 2024-03-29T04:10:06.323 INFO:teuthology.orchestra.run.smithi156.stderr: 52750K .......... .......... .......... .......... .......... 31% 27.4M 5s 2024-03-29T04:10:06.323 INFO:teuthology.orchestra.run.smithi156.stderr: 52800K .......... .......... .......... .......... .......... 31% 26.7M 5s 2024-03-29T04:10:06.325 INFO:teuthology.orchestra.run.smithi156.stderr: 52850K .......... .......... .......... .......... .......... 31% 24.0M 5s 2024-03-29T04:10:06.327 INFO:teuthology.orchestra.run.smithi156.stderr: 52900K .......... .......... .......... .......... .......... 31% 23.1M 5s 2024-03-29T04:10:06.328 INFO:teuthology.orchestra.run.smithi156.stderr: 52950K .......... .......... .......... .......... .......... 31% 27.1M 5s 2024-03-29T04:10:06.330 INFO:teuthology.orchestra.run.smithi156.stderr: 53000K .......... .......... .......... .......... .......... 31% 26.7M 5s 2024-03-29T04:10:06.332 INFO:teuthology.orchestra.run.smithi156.stderr: 53050K .......... .......... .......... .......... .......... 31% 27.5M 5s 2024-03-29T04:10:06.334 INFO:teuthology.orchestra.run.smithi156.stderr: 53100K .......... .......... .......... .......... .......... 31% 23.0M 5s 2024-03-29T04:10:06.336 INFO:teuthology.orchestra.run.smithi156.stderr: 53150K .......... .......... .......... .......... .......... 31% 24.8M 5s 2024-03-29T04:10:06.338 INFO:teuthology.orchestra.run.smithi156.stderr: 53200K .......... .......... .......... .......... .......... 31% 24.5M 5s 2024-03-29T04:10:06.341 INFO:teuthology.orchestra.run.smithi156.stderr: 53250K .......... .......... .......... .......... .......... 31% 23.5M 5s 2024-03-29T04:10:06.344 INFO:teuthology.orchestra.run.smithi156.stderr: 53300K .......... .......... .......... .......... .......... 31% 15.5M 5s 2024-03-29T04:10:06.345 INFO:teuthology.orchestra.run.smithi156.stderr: 53350K .......... .......... .......... .......... .......... 31% 24.9M 5s 2024-03-29T04:10:06.347 INFO:teuthology.orchestra.run.smithi156.stderr: 53400K .......... .......... .......... .......... .......... 31% 28.1M 5s 2024-03-29T04:10:06.349 INFO:teuthology.orchestra.run.smithi156.stderr: 53450K .......... .......... .......... .......... .......... 31% 27.8M 5s 2024-03-29T04:10:06.351 INFO:teuthology.orchestra.run.smithi156.stderr: 53500K .......... .......... .......... .......... .......... 31% 23.4M 5s 2024-03-29T04:10:06.352 INFO:teuthology.orchestra.run.smithi156.stderr: 53550K .......... .......... .......... .......... .......... 31% 36.8M 5s 2024-03-29T04:10:06.354 INFO:teuthology.orchestra.run.smithi156.stderr: 53600K .......... .......... .......... .......... .......... 31% 37.4M 5s 2024-03-29T04:10:06.355 INFO:teuthology.orchestra.run.smithi156.stderr: 53650K .......... .......... .......... .......... .......... 31% 36.0M 5s 2024-03-29T04:10:06.357 INFO:teuthology.orchestra.run.smithi156.stderr: 53700K .......... .......... .......... .......... .......... 31% 27.0M 5s 2024-03-29T04:10:06.358 INFO:teuthology.orchestra.run.smithi156.stderr: 53750K .......... .......... .......... .......... .......... 31% 26.6M 5s 2024-03-29T04:10:06.360 INFO:teuthology.orchestra.run.smithi156.stderr: 53800K .......... .......... .......... .......... .......... 31% 27.1M 5s 2024-03-29T04:10:06.362 INFO:teuthology.orchestra.run.smithi156.stderr: 53850K .......... .......... .......... .......... .......... 31% 27.7M 5s 2024-03-29T04:10:06.364 INFO:teuthology.orchestra.run.smithi156.stderr: 53900K .......... .......... .......... .......... .......... 32% 30.7M 5s 2024-03-29T04:10:06.365 INFO:teuthology.orchestra.run.smithi156.stderr: 53950K .......... .......... .......... .......... .......... 32% 35.2M 5s 2024-03-29T04:10:06.367 INFO:teuthology.orchestra.run.smithi156.stderr: 54000K .......... .......... .......... .......... .......... 32% 34.9M 5s 2024-03-29T04:10:06.368 INFO:teuthology.orchestra.run.smithi156.stderr: 54050K .......... .......... .......... .......... .......... 32% 36.2M 5s 2024-03-29T04:10:06.370 INFO:teuthology.orchestra.run.smithi156.stderr: 54100K .......... .......... .......... .......... .......... 32% 30.2M 5s 2024-03-29T04:10:06.371 INFO:teuthology.orchestra.run.smithi156.stderr: 54150K .......... .......... .......... .......... .......... 32% 34.3M 5s 2024-03-29T04:10:06.372 INFO:teuthology.orchestra.run.smithi156.stderr: 54200K .......... .......... .......... .......... .......... 32% 33.9M 5s 2024-03-29T04:10:06.374 INFO:teuthology.orchestra.run.smithi156.stderr: 54250K .......... .......... .......... .......... .......... 32% 34.6M 5s 2024-03-29T04:10:06.376 INFO:teuthology.orchestra.run.smithi156.stderr: 54300K .......... .......... .......... .......... .......... 32% 29.0M 5s 2024-03-29T04:10:06.377 INFO:teuthology.orchestra.run.smithi156.stderr: 54350K .......... .......... .......... .......... .......... 32% 34.7M 5s 2024-03-29T04:10:06.378 INFO:teuthology.orchestra.run.smithi156.stderr: 54400K .......... .......... .......... .......... .......... 32% 33.9M 5s 2024-03-29T04:10:06.380 INFO:teuthology.orchestra.run.smithi156.stderr: 54450K .......... .......... .......... .......... .......... 32% 33.1M 5s 2024-03-29T04:10:06.381 INFO:teuthology.orchestra.run.smithi156.stderr: 54500K .......... .......... .......... .......... .......... 32% 28.8M 5s 2024-03-29T04:10:06.383 INFO:teuthology.orchestra.run.smithi156.stderr: 54550K .......... .......... .......... .......... .......... 32% 34.7M 5s 2024-03-29T04:10:06.384 INFO:teuthology.orchestra.run.smithi156.stderr: 54600K .......... .......... .......... .......... .......... 32% 32.6M 5s 2024-03-29T04:10:06.386 INFO:teuthology.orchestra.run.smithi156.stderr: 54650K .......... .......... .......... .......... .......... 32% 33.1M 5s 2024-03-29T04:10:06.388 INFO:teuthology.orchestra.run.smithi156.stderr: 54700K .......... .......... .......... .......... .......... 32% 27.9M 5s 2024-03-29T04:10:06.389 INFO:teuthology.orchestra.run.smithi156.stderr: 54750K .......... .......... .......... .......... .......... 32% 32.5M 5s 2024-03-29T04:10:06.391 INFO:teuthology.orchestra.run.smithi156.stderr: 54800K .......... .......... .......... .......... .......... 32% 32.1M 5s 2024-03-29T04:10:06.392 INFO:teuthology.orchestra.run.smithi156.stderr: 54850K .......... .......... .......... .......... .......... 32% 31.5M 5s 2024-03-29T04:10:06.394 INFO:teuthology.orchestra.run.smithi156.stderr: 54900K .......... .......... .......... .......... .......... 32% 24.8M 5s 2024-03-29T04:10:06.396 INFO:teuthology.orchestra.run.smithi156.stderr: 54950K .......... .......... .......... .......... .......... 32% 28.6M 5s 2024-03-29T04:10:06.398 INFO:teuthology.orchestra.run.smithi156.stderr: 55000K .......... .......... .......... .......... .......... 32% 28.7M 5s 2024-03-29T04:10:06.400 INFO:teuthology.orchestra.run.smithi156.stderr: 55050K .......... .......... .......... .......... .......... 32% 26.8M 5s 2024-03-29T04:10:06.402 INFO:teuthology.orchestra.run.smithi156.stderr: 55100K .......... .......... .......... .......... .......... 32% 20.6M 5s 2024-03-29T04:10:06.404 INFO:teuthology.orchestra.run.smithi156.stderr: 55150K .......... .......... .......... .......... .......... 32% 25.1M 5s 2024-03-29T04:10:06.405 INFO:teuthology.orchestra.run.smithi156.stderr: 55200K .......... .......... .......... .......... .......... 32% 26.9M 5s 2024-03-29T04:10:06.407 INFO:teuthology.orchestra.run.smithi156.stderr: 55250K .......... .......... .......... .......... .......... 32% 26.5M 5s 2024-03-29T04:10:06.409 INFO:teuthology.orchestra.run.smithi156.stderr: 55300K .......... .......... .......... .......... .......... 32% 22.3M 5s 2024-03-29T04:10:06.411 INFO:teuthology.orchestra.run.smithi156.stderr: 55350K .......... .......... .......... .......... .......... 32% 25.9M 5s 2024-03-29T04:10:06.413 INFO:teuthology.orchestra.run.smithi156.stderr: 55400K .......... .......... .......... .......... .......... 32% 25.9M 5s 2024-03-29T04:10:06.415 INFO:teuthology.orchestra.run.smithi156.stderr: 55450K .......... .......... .......... .......... .......... 32% 25.8M 5s 2024-03-29T04:10:06.417 INFO:teuthology.orchestra.run.smithi156.stderr: 55500K .......... .......... .......... .......... .......... 32% 21.0M 5s 2024-03-29T04:10:06.419 INFO:teuthology.orchestra.run.smithi156.stderr: 55550K .......... .......... .......... .......... .......... 32% 28.5M 5s 2024-03-29T04:10:06.421 INFO:teuthology.orchestra.run.smithi156.stderr: 55600K .......... .......... .......... .......... .......... 33% 29.8M 5s 2024-03-29T04:10:06.423 INFO:teuthology.orchestra.run.smithi156.stderr: 55650K .......... .......... .......... .......... .......... 33% 28.6M 5s 2024-03-29T04:10:06.424 INFO:teuthology.orchestra.run.smithi156.stderr: 55700K .......... .......... .......... .......... .......... 33% 26.7M 5s 2024-03-29T04:10:06.426 INFO:teuthology.orchestra.run.smithi156.stderr: 55750K .......... .......... .......... .......... .......... 33% 33.1M 5s 2024-03-29T04:10:06.427 INFO:teuthology.orchestra.run.smithi156.stderr: 55800K .......... .......... .......... .......... .......... 33% 32.9M 5s 2024-03-29T04:10:06.429 INFO:teuthology.orchestra.run.smithi156.stderr: 55850K .......... .......... .......... .......... .......... 33% 33.1M 5s 2024-03-29T04:10:06.430 INFO:teuthology.orchestra.run.smithi156.stderr: 55900K .......... .......... .......... .......... .......... 33% 27.4M 5s 2024-03-29T04:10:06.432 INFO:teuthology.orchestra.run.smithi156.stderr: 55950K .......... .......... .......... .......... .......... 33% 30.9M 5s 2024-03-29T04:10:06.434 INFO:teuthology.orchestra.run.smithi156.stderr: 56000K .......... .......... .......... .......... .......... 33% 32.7M 4s 2024-03-29T04:10:06.435 INFO:teuthology.orchestra.run.smithi156.stderr: 56050K .......... .......... .......... .......... .......... 33% 29.0M 4s 2024-03-29T04:10:06.437 INFO:teuthology.orchestra.run.smithi156.stderr: 56100K .......... .......... .......... .......... .......... 33% 26.8M 4s 2024-03-29T04:10:06.439 INFO:teuthology.orchestra.run.smithi156.stderr: 56150K .......... .......... .......... .......... .......... 33% 31.9M 4s 2024-03-29T04:10:06.440 INFO:teuthology.orchestra.run.smithi156.stderr: 56200K .......... .......... .......... .......... .......... 33% 31.9M 4s 2024-03-29T04:10:06.442 INFO:teuthology.orchestra.run.smithi156.stderr: 56250K .......... .......... .......... .......... .......... 33% 30.9M 4s 2024-03-29T04:10:06.444 INFO:teuthology.orchestra.run.smithi156.stderr: 56300K .......... .......... .......... .......... .......... 33% 26.1M 4s 2024-03-29T04:10:06.445 INFO:teuthology.orchestra.run.smithi156.stderr: 56350K .......... .......... .......... .......... .......... 33% 30.7M 4s 2024-03-29T04:10:06.447 INFO:teuthology.orchestra.run.smithi156.stderr: 56400K .......... .......... .......... .......... .......... 33% 30.8M 4s 2024-03-29T04:10:06.449 INFO:teuthology.orchestra.run.smithi156.stderr: 56450K .......... .......... .......... .......... .......... 33% 31.4M 4s 2024-03-29T04:10:06.450 INFO:teuthology.orchestra.run.smithi156.stderr: 56500K .......... .......... .......... .......... .......... 33% 26.8M 4s 2024-03-29T04:10:06.452 INFO:teuthology.orchestra.run.smithi156.stderr: 56550K .......... .......... .......... .......... .......... 33% 29.9M 4s 2024-03-29T04:10:06.453 INFO:teuthology.orchestra.run.smithi156.stderr: 56600K .......... .......... .......... .......... .......... 33% 29.4M 4s 2024-03-29T04:10:06.455 INFO:teuthology.orchestra.run.smithi156.stderr: 56650K .......... .......... .......... .......... .......... 33% 31.2M 4s 2024-03-29T04:10:06.457 INFO:teuthology.orchestra.run.smithi156.stderr: 56700K .......... .......... .......... .......... .......... 33% 23.6M 4s 2024-03-29T04:10:06.459 INFO:teuthology.orchestra.run.smithi156.stderr: 56750K .......... .......... .......... .......... .......... 33% 30.3M 4s 2024-03-29T04:10:06.460 INFO:teuthology.orchestra.run.smithi156.stderr: 56800K .......... .......... .......... .......... .......... 33% 29.7M 4s 2024-03-29T04:10:06.462 INFO:teuthology.orchestra.run.smithi156.stderr: 56850K .......... .......... .......... .......... .......... 33% 29.2M 4s 2024-03-29T04:10:06.464 INFO:teuthology.orchestra.run.smithi156.stderr: 56900K .......... .......... .......... .......... .......... 33% 25.0M 4s 2024-03-29T04:10:06.466 INFO:teuthology.orchestra.run.smithi156.stderr: 56950K .......... .......... .......... .......... .......... 33% 29.2M 4s 2024-03-29T04:10:06.467 INFO:teuthology.orchestra.run.smithi156.stderr: 57000K .......... .......... .......... .......... .......... 33% 30.0M 4s 2024-03-29T04:10:06.469 INFO:teuthology.orchestra.run.smithi156.stderr: 57050K .......... .......... .......... .......... .......... 33% 29.0M 4s 2024-03-29T04:10:06.471 INFO:teuthology.orchestra.run.smithi156.stderr: 57100K .......... .......... .......... .......... .......... 33% 24.8M 4s 2024-03-29T04:10:06.473 INFO:teuthology.orchestra.run.smithi156.stderr: 57150K .......... .......... .......... .......... .......... 33% 25.7M 4s 2024-03-29T04:10:06.475 INFO:teuthology.orchestra.run.smithi156.stderr: 57200K .......... .......... .......... .......... .......... 33% 21.1M 4s 2024-03-29T04:10:06.478 INFO:teuthology.orchestra.run.smithi156.stderr: 57250K .......... .......... .......... .......... .......... 33% 21.1M 4s 2024-03-29T04:10:06.480 INFO:teuthology.orchestra.run.smithi156.stderr: 57300K .......... .......... .......... .......... .......... 34% 18.7M 4s 2024-03-29T04:10:06.483 INFO:teuthology.orchestra.run.smithi156.stderr: 57350K .......... .......... .......... .......... .......... 34% 19.7M 4s 2024-03-29T04:10:06.484 INFO:teuthology.orchestra.run.smithi156.stderr: 57400K .......... .......... .......... .......... .......... 34% 25.9M 4s 2024-03-29T04:10:06.487 INFO:teuthology.orchestra.run.smithi156.stderr: 57450K .......... .......... .......... .......... .......... 34% 24.0M 4s 2024-03-29T04:10:06.489 INFO:teuthology.orchestra.run.smithi156.stderr: 57500K .......... .......... .......... .......... .......... 34% 22.6M 4s 2024-03-29T04:10:06.490 INFO:teuthology.orchestra.run.smithi156.stderr: 57550K .......... .......... .......... .......... .......... 34% 26.1M 4s 2024-03-29T04:10:06.492 INFO:teuthology.orchestra.run.smithi156.stderr: 57600K .......... .......... .......... .......... .......... 34% 30.6M 4s 2024-03-29T04:10:06.494 INFO:teuthology.orchestra.run.smithi156.stderr: 57650K .......... .......... .......... .......... .......... 34% 27.3M 4s 2024-03-29T04:10:06.496 INFO:teuthology.orchestra.run.smithi156.stderr: 57700K .......... .......... .......... .......... .......... 34% 29.4M 4s 2024-03-29T04:10:06.497 INFO:teuthology.orchestra.run.smithi156.stderr: 57750K .......... .......... .......... .......... .......... 34% 34.6M 4s 2024-03-29T04:10:06.498 INFO:teuthology.orchestra.run.smithi156.stderr: 57800K .......... .......... .......... .......... .......... 34% 38.9M 4s 2024-03-29T04:10:06.500 INFO:teuthology.orchestra.run.smithi156.stderr: 57850K .......... .......... .......... .......... .......... 34% 39.2M 4s 2024-03-29T04:10:06.501 INFO:teuthology.orchestra.run.smithi156.stderr: 57900K .......... .......... .......... .......... .......... 34% 33.7M 4s 2024-03-29T04:10:06.502 INFO:teuthology.orchestra.run.smithi156.stderr: 57950K .......... .......... .......... .......... .......... 34% 40.3M 4s 2024-03-29T04:10:06.503 INFO:teuthology.orchestra.run.smithi156.stderr: 58000K .......... .......... .......... .......... .......... 34% 39.3M 4s 2024-03-29T04:10:06.505 INFO:teuthology.orchestra.run.smithi156.stderr: 58050K .......... .......... .......... .......... .......... 34% 39.5M 4s 2024-03-29T04:10:06.506 INFO:teuthology.orchestra.run.smithi156.stderr: 58100K .......... .......... .......... .......... .......... 34% 33.5M 4s 2024-03-29T04:10:06.507 INFO:teuthology.orchestra.run.smithi156.stderr: 58150K .......... .......... .......... .......... .......... 34% 39.1M 4s 2024-03-29T04:10:06.509 INFO:teuthology.orchestra.run.smithi156.stderr: 58200K .......... .......... .......... .......... .......... 34% 38.4M 4s 2024-03-29T04:10:06.510 INFO:teuthology.orchestra.run.smithi156.stderr: 58250K .......... .......... .......... .......... .......... 34% 37.2M 4s 2024-03-29T04:10:06.511 INFO:teuthology.orchestra.run.smithi156.stderr: 58300K .......... .......... .......... .......... .......... 34% 30.2M 4s 2024-03-29T04:10:06.513 INFO:teuthology.orchestra.run.smithi156.stderr: 58350K .......... .......... .......... .......... .......... 34% 36.8M 4s 2024-03-29T04:10:06.514 INFO:teuthology.orchestra.run.smithi156.stderr: 58400K .......... .......... .......... .......... .......... 34% 37.2M 4s 2024-03-29T04:10:06.515 INFO:teuthology.orchestra.run.smithi156.stderr: 58450K .......... .......... .......... .......... .......... 34% 37.4M 4s 2024-03-29T04:10:06.517 INFO:teuthology.orchestra.run.smithi156.stderr: 58500K .......... .......... .......... .......... .......... 34% 31.8M 4s 2024-03-29T04:10:06.518 INFO:teuthology.orchestra.run.smithi156.stderr: 58550K .......... .......... .......... .......... .......... 34% 36.7M 4s 2024-03-29T04:10:06.519 INFO:teuthology.orchestra.run.smithi156.stderr: 58600K .......... .......... .......... .......... .......... 34% 37.7M 4s 2024-03-29T04:10:06.521 INFO:teuthology.orchestra.run.smithi156.stderr: 58650K .......... .......... .......... .......... .......... 34% 37.9M 4s 2024-03-29T04:10:06.522 INFO:teuthology.orchestra.run.smithi156.stderr: 58700K .......... .......... .......... .......... .......... 34% 31.6M 4s 2024-03-29T04:10:06.524 INFO:teuthology.orchestra.run.smithi156.stderr: 58750K .......... .......... .......... .......... .......... 34% 38.1M 4s 2024-03-29T04:10:06.525 INFO:teuthology.orchestra.run.smithi156.stderr: 58800K .......... .......... .......... .......... .......... 34% 38.4M 4s 2024-03-29T04:10:06.526 INFO:teuthology.orchestra.run.smithi156.stderr: 58850K .......... .......... .......... .......... .......... 34% 38.2M 4s 2024-03-29T04:10:06.531 INFO:teuthology.orchestra.run.smithi156.stderr: 58900K .......... .......... .......... .......... .......... 34% 32.8M 4s 2024-03-29T04:10:06.531 INFO:teuthology.orchestra.run.smithi156.stderr: 58950K .......... .......... .......... .......... .......... 35% 36.1M 4s 2024-03-29T04:10:06.531 INFO:teuthology.orchestra.run.smithi156.stderr: 59000K .......... .......... .......... .......... .......... 35% 36.7M 4s 2024-03-29T04:10:06.532 INFO:teuthology.orchestra.run.smithi156.stderr: 59050K .......... .......... .......... .......... .......... 35% 37.6M 4s 2024-03-29T04:10:06.533 INFO:teuthology.orchestra.run.smithi156.stderr: 59100K .......... .......... .......... .......... .......... 35% 32.1M 4s 2024-03-29T04:10:06.534 INFO:teuthology.orchestra.run.smithi156.stderr: 59150K .......... .......... .......... .......... .......... 35% 37.3M 4s 2024-03-29T04:10:06.536 INFO:teuthology.orchestra.run.smithi156.stderr: 59200K .......... .......... .......... .......... .......... 35% 37.1M 4s 2024-03-29T04:10:06.538 INFO:teuthology.orchestra.run.smithi156.stderr: 59250K .......... .......... .......... .......... .......... 35% 36.6M 4s 2024-03-29T04:10:06.539 INFO:teuthology.orchestra.run.smithi156.stderr: 59300K .......... .......... .......... .......... .......... 35% 31.1M 4s 2024-03-29T04:10:06.540 INFO:teuthology.orchestra.run.smithi156.stderr: 59350K .......... .......... .......... .......... .......... 35% 36.4M 4s 2024-03-29T04:10:06.542 INFO:teuthology.orchestra.run.smithi156.stderr: 59400K .......... .......... .......... .......... .......... 35% 32.5M 4s 2024-03-29T04:10:06.543 INFO:teuthology.orchestra.run.smithi156.stderr: 59450K .......... .......... .......... .......... .......... 35% 34.3M 4s 2024-03-29T04:10:06.545 INFO:teuthology.orchestra.run.smithi156.stderr: 59500K .......... .......... .......... .......... .......... 35% 30.9M 4s 2024-03-29T04:10:06.546 INFO:teuthology.orchestra.run.smithi156.stderr: 59550K .......... .......... .......... .......... .......... 35% 31.1M 4s 2024-03-29T04:10:06.548 INFO:teuthology.orchestra.run.smithi156.stderr: 59600K .......... .......... .......... .......... .......... 35% 29.6M 4s 2024-03-29T04:10:06.549 INFO:teuthology.orchestra.run.smithi156.stderr: 59650K .......... .......... .......... .......... .......... 35% 31.2M 4s 2024-03-29T04:10:06.551 INFO:teuthology.orchestra.run.smithi156.stderr: 59700K .......... .......... .......... .......... .......... 35% 26.8M 4s 2024-03-29T04:10:06.553 INFO:teuthology.orchestra.run.smithi156.stderr: 59750K .......... .......... .......... .......... .......... 35% 31.5M 4s 2024-03-29T04:10:06.554 INFO:teuthology.orchestra.run.smithi156.stderr: 59800K .......... .......... .......... .......... .......... 35% 31.2M 4s 2024-03-29T04:10:06.556 INFO:teuthology.orchestra.run.smithi156.stderr: 59850K .......... .......... .......... .......... .......... 35% 31.1M 4s 2024-03-29T04:10:06.558 INFO:teuthology.orchestra.run.smithi156.stderr: 59900K .......... .......... .......... .......... .......... 35% 26.5M 4s 2024-03-29T04:10:06.559 INFO:teuthology.orchestra.run.smithi156.stderr: 59950K .......... .......... .......... .......... .......... 35% 30.6M 4s 2024-03-29T04:10:06.561 INFO:teuthology.orchestra.run.smithi156.stderr: 60000K .......... .......... .......... .......... .......... 35% 30.5M 4s 2024-03-29T04:10:06.563 INFO:teuthology.orchestra.run.smithi156.stderr: 60050K .......... .......... .......... .......... .......... 35% 30.3M 4s 2024-03-29T04:10:06.564 INFO:teuthology.orchestra.run.smithi156.stderr: 60100K .......... .......... .......... .......... .......... 35% 25.7M 4s 2024-03-29T04:10:06.566 INFO:teuthology.orchestra.run.smithi156.stderr: 60150K .......... .......... .......... .......... .......... 35% 30.4M 4s 2024-03-29T04:10:06.568 INFO:teuthology.orchestra.run.smithi156.stderr: 60200K .......... .......... .......... .......... .......... 35% 30.3M 4s 2024-03-29T04:10:06.570 INFO:teuthology.orchestra.run.smithi156.stderr: 60250K .......... .......... .......... .......... .......... 35% 29.5M 4s 2024-03-29T04:10:06.571 INFO:teuthology.orchestra.run.smithi156.stderr: 60300K .......... .......... .......... .......... .......... 35% 25.5M 4s 2024-03-29T04:10:06.573 INFO:teuthology.orchestra.run.smithi156.stderr: 60350K .......... .......... .......... .......... .......... 35% 29.7M 4s 2024-03-29T04:10:06.574 INFO:teuthology.orchestra.run.smithi156.stderr: 60400K .......... .......... .......... .......... .......... 35% 30.1M 4s 2024-03-29T04:10:06.578 INFO:teuthology.orchestra.run.smithi156.stderr: 60450K .......... .......... .......... .......... .......... 35% 28.8M 4s 2024-03-29T04:10:06.578 INFO:teuthology.orchestra.run.smithi156.stderr: 60500K .......... .......... .......... .......... .......... 35% 25.6M 4s 2024-03-29T04:10:06.580 INFO:teuthology.orchestra.run.smithi156.stderr: 60550K .......... .......... .......... .......... .......... 35% 29.2M 4s 2024-03-29T04:10:06.581 INFO:teuthology.orchestra.run.smithi156.stderr: 60600K .......... .......... .......... .......... .......... 35% 29.5M 4s 2024-03-29T04:10:06.583 INFO:teuthology.orchestra.run.smithi156.stderr: 60650K .......... .......... .......... .......... .......... 36% 28.7M 4s 2024-03-29T04:10:06.585 INFO:teuthology.orchestra.run.smithi156.stderr: 60700K .......... .......... .......... .......... .......... 36% 25.1M 4s 2024-03-29T04:10:06.587 INFO:teuthology.orchestra.run.smithi156.stderr: 60750K .......... .......... .......... .......... .......... 36% 29.0M 4s 2024-03-29T04:10:06.589 INFO:teuthology.orchestra.run.smithi156.stderr: 60800K .......... .......... .......... .......... .......... 36% 28.2M 4s 2024-03-29T04:10:06.590 INFO:teuthology.orchestra.run.smithi156.stderr: 60850K .......... .......... .......... .......... .......... 36% 28.2M 4s 2024-03-29T04:10:06.592 INFO:teuthology.orchestra.run.smithi156.stderr: 60900K .......... .......... .......... .......... .......... 36% 23.5M 4s 2024-03-29T04:10:06.594 INFO:teuthology.orchestra.run.smithi156.stderr: 60950K .......... .......... .......... .......... .......... 36% 27.8M 4s 2024-03-29T04:10:06.596 INFO:teuthology.orchestra.run.smithi156.stderr: 61000K .......... .......... .......... .......... .......... 36% 27.8M 4s 2024-03-29T04:10:06.598 INFO:teuthology.orchestra.run.smithi156.stderr: 61050K .......... .......... .......... .......... .......... 36% 27.9M 4s 2024-03-29T04:10:06.600 INFO:teuthology.orchestra.run.smithi156.stderr: 61100K .......... .......... .......... .......... .......... 36% 23.3M 4s 2024-03-29T04:10:06.601 INFO:teuthology.orchestra.run.smithi156.stderr: 61150K .......... .......... .......... .......... .......... 36% 27.9M 4s 2024-03-29T04:10:06.603 INFO:teuthology.orchestra.run.smithi156.stderr: 61200K .......... .......... .......... .......... .......... 36% 27.4M 4s 2024-03-29T04:10:06.605 INFO:teuthology.orchestra.run.smithi156.stderr: 61250K .......... .......... .......... .......... .......... 36% 27.0M 4s 2024-03-29T04:10:06.607 INFO:teuthology.orchestra.run.smithi156.stderr: 61300K .......... .......... .......... .......... .......... 36% 23.3M 4s 2024-03-29T04:10:06.609 INFO:teuthology.orchestra.run.smithi156.stderr: 61350K .......... .......... .......... .......... .......... 36% 26.6M 4s 2024-03-29T04:10:06.611 INFO:teuthology.orchestra.run.smithi156.stderr: 61400K .......... .......... .......... .......... .......... 36% 28.6M 4s 2024-03-29T04:10:06.612 INFO:teuthology.orchestra.run.smithi156.stderr: 61450K .......... .......... .......... .......... .......... 36% 33.5M 4s 2024-03-29T04:10:06.614 INFO:teuthology.orchestra.run.smithi156.stderr: 61500K .......... .......... .......... .......... .......... 36% 25.8M 4s 2024-03-29T04:10:06.616 INFO:teuthology.orchestra.run.smithi156.stderr: 61550K .......... .......... .......... .......... .......... 36% 29.9M 4s 2024-03-29T04:10:06.617 INFO:teuthology.orchestra.run.smithi156.stderr: 61600K .......... .......... .......... .......... .......... 36% 29.3M 4s 2024-03-29T04:10:06.620 INFO:teuthology.orchestra.run.smithi156.stderr: 61650K .......... .......... .......... .......... .......... 36% 24.9M 4s 2024-03-29T04:10:06.622 INFO:teuthology.orchestra.run.smithi156.stderr: 61700K .......... .......... .......... .......... .......... 36% 20.5M 4s 2024-03-29T04:10:06.623 INFO:teuthology.orchestra.run.smithi156.stderr: 61750K .......... .......... .......... .......... .......... 36% 30.0M 4s 2024-03-29T04:10:06.625 INFO:teuthology.orchestra.run.smithi156.stderr: 61800K .......... .......... .......... .......... .......... 36% 29.3M 4s 2024-03-29T04:10:06.627 INFO:teuthology.orchestra.run.smithi156.stderr: 61850K .......... .......... .......... .......... .......... 36% 28.2M 4s 2024-03-29T04:10:06.628 INFO:teuthology.orchestra.run.smithi156.stderr: 61900K .......... .......... .......... .......... .......... 36% 29.9M 4s 2024-03-29T04:10:06.629 INFO:teuthology.orchestra.run.smithi156.stderr: 61950K .......... .......... .......... .......... .......... 36% 39.8M 4s 2024-03-29T04:10:06.631 INFO:teuthology.orchestra.run.smithi156.stderr: 62000K .......... .......... .......... .......... .......... 36% 40.8M 4s 2024-03-29T04:10:06.632 INFO:teuthology.orchestra.run.smithi156.stderr: 62050K .......... .......... .......... .......... .......... 36% 40.8M 4s 2024-03-29T04:10:06.633 INFO:teuthology.orchestra.run.smithi156.stderr: 62100K .......... .......... .......... .......... .......... 36% 35.2M 4s 2024-03-29T04:10:06.635 INFO:teuthology.orchestra.run.smithi156.stderr: 62150K .......... .......... .......... .......... .......... 36% 38.7M 4s 2024-03-29T04:10:06.636 INFO:teuthology.orchestra.run.smithi156.stderr: 62200K .......... .......... .......... .......... .......... 36% 40.5M 4s 2024-03-29T04:10:06.637 INFO:teuthology.orchestra.run.smithi156.stderr: 62250K .......... .......... .......... .......... .......... 36% 40.8M 4s 2024-03-29T04:10:06.639 INFO:teuthology.orchestra.run.smithi156.stderr: 62300K .......... .......... .......... .......... .......... 36% 28.8M 4s 2024-03-29T04:10:06.641 INFO:teuthology.orchestra.run.smithi156.stderr: 62350K .......... .......... .......... .......... .......... 37% 24.8M 4s 2024-03-29T04:10:06.643 INFO:teuthology.orchestra.run.smithi156.stderr: 62400K .......... .......... .......... .......... .......... 37% 24.3M 4s 2024-03-29T04:10:06.645 INFO:teuthology.orchestra.run.smithi156.stderr: 62450K .......... .......... .......... .......... .......... 37% 25.7M 4s 2024-03-29T04:10:06.647 INFO:teuthology.orchestra.run.smithi156.stderr: 62500K .......... .......... .......... .......... .......... 37% 22.4M 4s 2024-03-29T04:10:06.649 INFO:teuthology.orchestra.run.smithi156.stderr: 62550K .......... .......... .......... .......... .......... 37% 26.6M 4s 2024-03-29T04:10:06.650 INFO:teuthology.orchestra.run.smithi156.stderr: 62600K .......... .......... .......... .......... .......... 37% 26.3M 4s 2024-03-29T04:10:06.652 INFO:teuthology.orchestra.run.smithi156.stderr: 62650K .......... .......... .......... .......... .......... 37% 25.6M 4s 2024-03-29T04:10:06.654 INFO:teuthology.orchestra.run.smithi156.stderr: 62700K .......... .......... .......... .......... .......... 37% 23.3M 4s 2024-03-29T04:10:06.656 INFO:teuthology.orchestra.run.smithi156.stderr: 62750K .......... .......... .......... .......... .......... 37% 25.6M 4s 2024-03-29T04:10:06.659 INFO:teuthology.orchestra.run.smithi156.stderr: 62800K .......... .......... .......... .......... .......... 37% 27.0M 4s 2024-03-29T04:10:06.660 INFO:teuthology.orchestra.run.smithi156.stderr: 62850K .......... .......... .......... .......... .......... 37% 26.0M 4s 2024-03-29T04:10:06.662 INFO:teuthology.orchestra.run.smithi156.stderr: 62900K .......... .......... .......... .......... .......... 37% 22.6M 4s 2024-03-29T04:10:06.664 INFO:teuthology.orchestra.run.smithi156.stderr: 62950K .......... .......... .......... .......... .......... 37% 26.2M 4s 2024-03-29T04:10:06.666 INFO:teuthology.orchestra.run.smithi156.stderr: 63000K .......... .......... .......... .......... .......... 37% 27.8M 4s 2024-03-29T04:10:06.668 INFO:teuthology.orchestra.run.smithi156.stderr: 63050K .......... .......... .......... .......... .......... 37% 28.5M 4s 2024-03-29T04:10:06.670 INFO:teuthology.orchestra.run.smithi156.stderr: 63100K .......... .......... .......... .......... .......... 37% 24.6M 4s 2024-03-29T04:10:06.671 INFO:teuthology.orchestra.run.smithi156.stderr: 63150K .......... .......... .......... .......... .......... 37% 27.7M 4s 2024-03-29T04:10:06.673 INFO:teuthology.orchestra.run.smithi156.stderr: 63200K .......... .......... .......... .......... .......... 37% 28.0M 4s 2024-03-29T04:10:06.675 INFO:teuthology.orchestra.run.smithi156.stderr: 63250K .......... .......... .......... .......... .......... 37% 28.6M 4s 2024-03-29T04:10:06.677 INFO:teuthology.orchestra.run.smithi156.stderr: 63300K .......... .......... .......... .......... .......... 37% 24.0M 4s 2024-03-29T04:10:06.678 INFO:teuthology.orchestra.run.smithi156.stderr: 63350K .......... .......... .......... .......... .......... 37% 27.4M 4s 2024-03-29T04:10:06.680 INFO:teuthology.orchestra.run.smithi156.stderr: 63400K .......... .......... .......... .......... .......... 37% 27.6M 4s 2024-03-29T04:10:06.682 INFO:teuthology.orchestra.run.smithi156.stderr: 63450K .......... .......... .......... .......... .......... 37% 27.5M 4s 2024-03-29T04:10:06.684 INFO:teuthology.orchestra.run.smithi156.stderr: 63500K .......... .......... .......... .......... .......... 37% 23.5M 4s 2024-03-29T04:10:06.686 INFO:teuthology.orchestra.run.smithi156.stderr: 63550K .......... .......... .......... .......... .......... 37% 27.1M 4s 2024-03-29T04:10:06.688 INFO:teuthology.orchestra.run.smithi156.stderr: 63600K .......... .......... .......... .......... .......... 37% 27.0M 4s 2024-03-29T04:10:06.690 INFO:teuthology.orchestra.run.smithi156.stderr: 63650K .......... .......... .......... .......... .......... 37% 27.5M 4s 2024-03-29T04:10:06.692 INFO:teuthology.orchestra.run.smithi156.stderr: 63700K .......... .......... .......... .......... .......... 37% 22.7M 4s 2024-03-29T04:10:06.693 INFO:teuthology.orchestra.run.smithi156.stderr: 63750K .......... .......... .......... .......... .......... 37% 28.2M 4s 2024-03-29T04:10:06.695 INFO:teuthology.orchestra.run.smithi156.stderr: 63800K .......... .......... .......... .......... .......... 37% 29.5M 4s 2024-03-29T04:10:06.697 INFO:teuthology.orchestra.run.smithi156.stderr: 63850K .......... .......... .......... .......... .......... 37% 30.2M 4s 2024-03-29T04:10:06.699 INFO:teuthology.orchestra.run.smithi156.stderr: 63900K .......... .......... .......... .......... .......... 37% 25.6M 4s 2024-03-29T04:10:06.700 INFO:teuthology.orchestra.run.smithi156.stderr: 63950K .......... .......... .......... .......... .......... 37% 28.9M 4s 2024-03-29T04:10:06.702 INFO:teuthology.orchestra.run.smithi156.stderr: 64000K .......... .......... .......... .......... .......... 37% 29.8M 4s 2024-03-29T04:10:06.704 INFO:teuthology.orchestra.run.smithi156.stderr: 64050K .......... .......... .......... .......... .......... 38% 29.2M 4s 2024-03-29T04:10:06.706 INFO:teuthology.orchestra.run.smithi156.stderr: 64100K .......... .......... .......... .......... .......... 38% 24.5M 4s 2024-03-29T04:10:06.707 INFO:teuthology.orchestra.run.smithi156.stderr: 64150K .......... .......... .......... .......... .......... 38% 29.0M 4s 2024-03-29T04:10:06.709 INFO:teuthology.orchestra.run.smithi156.stderr: 64200K .......... .......... .......... .......... .......... 38% 28.8M 4s 2024-03-29T04:10:06.711 INFO:teuthology.orchestra.run.smithi156.stderr: 64250K .......... .......... .......... .......... .......... 38% 29.2M 4s 2024-03-29T04:10:06.713 INFO:teuthology.orchestra.run.smithi156.stderr: 64300K .......... .......... .......... .......... .......... 38% 24.2M 4s 2024-03-29T04:10:06.714 INFO:teuthology.orchestra.run.smithi156.stderr: 64350K .......... .......... .......... .......... .......... 38% 27.5M 4s 2024-03-29T04:10:06.716 INFO:teuthology.orchestra.run.smithi156.stderr: 64400K .......... .......... .......... .......... .......... 38% 28.4M 4s 2024-03-29T04:10:06.718 INFO:teuthology.orchestra.run.smithi156.stderr: 64450K .......... .......... .......... .......... .......... 38% 28.2M 4s 2024-03-29T04:10:06.720 INFO:teuthology.orchestra.run.smithi156.stderr: 64500K .......... .......... .......... .......... .......... 38% 23.6M 4s 2024-03-29T04:10:06.722 INFO:teuthology.orchestra.run.smithi156.stderr: 64550K .......... .......... .......... .......... .......... 38% 27.4M 4s 2024-03-29T04:10:06.724 INFO:teuthology.orchestra.run.smithi156.stderr: 64600K .......... .......... .......... .......... .......... 38% 25.5M 4s 2024-03-29T04:10:06.726 INFO:teuthology.orchestra.run.smithi156.stderr: 64650K .......... .......... .......... .......... .......... 38% 24.7M 4s 2024-03-29T04:10:06.730 INFO:teuthology.orchestra.run.smithi156.stderr: 64700K .......... .......... .......... .......... .......... 38% 21.3M 4s 2024-03-29T04:10:06.731 INFO:teuthology.orchestra.run.smithi156.stderr: 64750K .......... .......... .......... .......... .......... 38% 25.2M 4s 2024-03-29T04:10:06.732 INFO:teuthology.orchestra.run.smithi156.stderr: 64800K .......... .......... .......... .......... .......... 38% 25.2M 4s 2024-03-29T04:10:06.734 INFO:teuthology.orchestra.run.smithi156.stderr: 64850K .......... .......... .......... .......... .......... 38% 25.0M 4s 2024-03-29T04:10:06.736 INFO:teuthology.orchestra.run.smithi156.stderr: 64900K .......... .......... .......... .......... .......... 38% 21.1M 4s 2024-03-29T04:10:06.738 INFO:teuthology.orchestra.run.smithi156.stderr: 64950K .......... .......... .......... .......... .......... 38% 24.8M 4s 2024-03-29T04:10:06.740 INFO:teuthology.orchestra.run.smithi156.stderr: 65000K .......... .......... .......... .......... .......... 38% 24.6M 4s 2024-03-29T04:10:06.743 INFO:teuthology.orchestra.run.smithi156.stderr: 65050K .......... .......... .......... .......... .......... 38% 24.6M 4s 2024-03-29T04:10:06.744 INFO:teuthology.orchestra.run.smithi156.stderr: 65100K .......... .......... .......... .......... .......... 38% 21.2M 4s 2024-03-29T04:10:06.746 INFO:teuthology.orchestra.run.smithi156.stderr: 65150K .......... .......... .......... .......... .......... 38% 24.0M 4s 2024-03-29T04:10:06.748 INFO:teuthology.orchestra.run.smithi156.stderr: 65200K .......... .......... .......... .......... .......... 38% 24.2M 4s 2024-03-29T04:10:06.751 INFO:teuthology.orchestra.run.smithi156.stderr: 65250K .......... .......... .......... .......... .......... 38% 23.3M 4s 2024-03-29T04:10:06.753 INFO:teuthology.orchestra.run.smithi156.stderr: 65300K .......... .......... .......... .......... .......... 38% 19.1M 4s 2024-03-29T04:10:06.755 INFO:teuthology.orchestra.run.smithi156.stderr: 65350K .......... .......... .......... .......... .......... 38% 20.9M 4s 2024-03-29T04:10:06.758 INFO:teuthology.orchestra.run.smithi156.stderr: 65400K .......... .......... .......... .......... .......... 38% 22.1M 4s 2024-03-29T04:10:06.760 INFO:teuthology.orchestra.run.smithi156.stderr: 65450K .......... .......... .......... .......... .......... 38% 23.0M 4s 2024-03-29T04:10:06.762 INFO:teuthology.orchestra.run.smithi156.stderr: 65500K .......... .......... .......... .......... .......... 38% 17.7M 4s 2024-03-29T04:10:06.764 INFO:teuthology.orchestra.run.smithi156.stderr: 65550K .......... .......... .......... .......... .......... 38% 38.7M 4s 2024-03-29T04:10:06.765 INFO:teuthology.orchestra.run.smithi156.stderr: 65600K .......... .......... .......... .......... .......... 38% 35.6M 4s 2024-03-29T04:10:06.766 INFO:teuthology.orchestra.run.smithi156.stderr: 65650K .......... .......... .......... .......... .......... 38% 36.0M 4s 2024-03-29T04:10:06.768 INFO:teuthology.orchestra.run.smithi156.stderr: 65700K .......... .......... .......... .......... .......... 39% 26.0M 4s 2024-03-29T04:10:06.770 INFO:teuthology.orchestra.run.smithi156.stderr: 65750K .......... .......... .......... .......... .......... 39% 24.1M 4s 2024-03-29T04:10:06.772 INFO:teuthology.orchestra.run.smithi156.stderr: 65800K .......... .......... .......... .......... .......... 39% 26.5M 4s 2024-03-29T04:10:06.774 INFO:teuthology.orchestra.run.smithi156.stderr: 65850K .......... .......... .......... .......... .......... 39% 28.4M 4s 2024-03-29T04:10:06.776 INFO:teuthology.orchestra.run.smithi156.stderr: 65900K .......... .......... .......... .......... .......... 39% 26.5M 4s 2024-03-29T04:10:06.777 INFO:teuthology.orchestra.run.smithi156.stderr: 65950K .......... .......... .......... .......... .......... 39% 35.5M 4s 2024-03-29T04:10:06.779 INFO:teuthology.orchestra.run.smithi156.stderr: 66000K .......... .......... .......... .......... .......... 39% 32.0M 4s 2024-03-29T04:10:06.780 INFO:teuthology.orchestra.run.smithi156.stderr: 66050K .......... .......... .......... .......... .......... 39% 30.4M 4s 2024-03-29T04:10:06.782 INFO:teuthology.orchestra.run.smithi156.stderr: 66100K .......... .......... .......... .......... .......... 39% 33.8M 4s 2024-03-29T04:10:06.783 INFO:teuthology.orchestra.run.smithi156.stderr: 66150K .......... .......... .......... .......... .......... 39% 37.9M 4s 2024-03-29T04:10:06.784 INFO:teuthology.orchestra.run.smithi156.stderr: 66200K .......... .......... .......... .......... .......... 39% 38.9M 4s 2024-03-29T04:10:06.786 INFO:teuthology.orchestra.run.smithi156.stderr: 66250K .......... .......... .......... .......... .......... 39% 39.2M 4s 2024-03-29T04:10:06.787 INFO:teuthology.orchestra.run.smithi156.stderr: 66300K .......... .......... .......... .......... .......... 39% 33.0M 4s 2024-03-29T04:10:06.788 INFO:teuthology.orchestra.run.smithi156.stderr: 66350K .......... .......... .......... .......... .......... 39% 39.6M 4s 2024-03-29T04:10:06.789 INFO:teuthology.orchestra.run.smithi156.stderr: 66400K .......... .......... .......... .......... .......... 39% 39.6M 4s 2024-03-29T04:10:06.791 INFO:teuthology.orchestra.run.smithi156.stderr: 66450K .......... .......... .......... .......... .......... 39% 39.7M 4s 2024-03-29T04:10:06.792 INFO:teuthology.orchestra.run.smithi156.stderr: 66500K .......... .......... .......... .......... .......... 39% 33.1M 4s 2024-03-29T04:10:06.794 INFO:teuthology.orchestra.run.smithi156.stderr: 66550K .......... .......... .......... .......... .......... 39% 39.3M 4s 2024-03-29T04:10:06.795 INFO:teuthology.orchestra.run.smithi156.stderr: 66600K .......... .......... .......... .......... .......... 39% 37.4M 4s 2024-03-29T04:10:06.796 INFO:teuthology.orchestra.run.smithi156.stderr: 66650K .......... .......... .......... .......... .......... 39% 36.9M 4s 2024-03-29T04:10:06.797 INFO:teuthology.orchestra.run.smithi156.stderr: 66700K .......... .......... .......... .......... .......... 39% 33.4M 4s 2024-03-29T04:10:06.799 INFO:teuthology.orchestra.run.smithi156.stderr: 66750K .......... .......... .......... .......... .......... 39% 38.5M 4s 2024-03-29T04:10:06.800 INFO:teuthology.orchestra.run.smithi156.stderr: 66800K .......... .......... .......... .......... .......... 39% 37.1M 4s 2024-03-29T04:10:06.801 INFO:teuthology.orchestra.run.smithi156.stderr: 66850K .......... .......... .......... .......... .......... 39% 35.8M 4s 2024-03-29T04:10:06.803 INFO:teuthology.orchestra.run.smithi156.stderr: 66900K .......... .......... .......... .......... .......... 39% 31.7M 4s 2024-03-29T04:10:06.804 INFO:teuthology.orchestra.run.smithi156.stderr: 66950K .......... .......... .......... .......... .......... 39% 37.7M 4s 2024-03-29T04:10:06.806 INFO:teuthology.orchestra.run.smithi156.stderr: 67000K .......... .......... .......... .......... .......... 39% 34.1M 4s 2024-03-29T04:10:06.808 INFO:teuthology.orchestra.run.smithi156.stderr: 67050K .......... .......... .......... .......... .......... 39% 28.7M 4s 2024-03-29T04:10:06.809 INFO:teuthology.orchestra.run.smithi156.stderr: 67100K .......... .......... .......... .......... .......... 39% 25.9M 4s 2024-03-29T04:10:06.811 INFO:teuthology.orchestra.run.smithi156.stderr: 67150K .......... .......... .......... .......... .......... 39% 31.8M 4s 2024-03-29T04:10:06.812 INFO:teuthology.orchestra.run.smithi156.stderr: 67200K .......... .......... .......... .......... .......... 39% 33.7M 4s 2024-03-29T04:10:06.814 INFO:teuthology.orchestra.run.smithi156.stderr: 67250K .......... .......... .......... .......... .......... 39% 35.7M 4s 2024-03-29T04:10:06.815 INFO:teuthology.orchestra.run.smithi156.stderr: 67300K .......... .......... .......... .......... .......... 39% 29.0M 4s 2024-03-29T04:10:06.817 INFO:teuthology.orchestra.run.smithi156.stderr: 67350K .......... .......... .......... .......... .......... 39% 33.2M 4s 2024-03-29T04:10:06.818 INFO:teuthology.orchestra.run.smithi156.stderr: 67400K .......... .......... .......... .......... .......... 40% 33.9M 4s 2024-03-29T04:10:06.820 INFO:teuthology.orchestra.run.smithi156.stderr: 67450K .......... .......... .......... .......... .......... 40% 35.8M 4s 2024-03-29T04:10:06.821 INFO:teuthology.orchestra.run.smithi156.stderr: 67500K .......... .......... .......... .......... .......... 40% 28.6M 4s 2024-03-29T04:10:06.823 INFO:teuthology.orchestra.run.smithi156.stderr: 67550K .......... .......... .......... .......... .......... 40% 34.9M 4s 2024-03-29T04:10:06.824 INFO:teuthology.orchestra.run.smithi156.stderr: 67600K .......... .......... .......... .......... .......... 40% 35.9M 4s 2024-03-29T04:10:06.826 INFO:teuthology.orchestra.run.smithi156.stderr: 67650K .......... .......... .......... .......... .......... 40% 33.1M 4s 2024-03-29T04:10:06.827 INFO:teuthology.orchestra.run.smithi156.stderr: 67700K .......... .......... .......... .......... .......... 40% 28.3M 4s 2024-03-29T04:10:06.829 INFO:teuthology.orchestra.run.smithi156.stderr: 67750K .......... .......... .......... .......... .......... 40% 32.7M 4s 2024-03-29T04:10:06.830 INFO:teuthology.orchestra.run.smithi156.stderr: 67800K .......... .......... .......... .......... .......... 40% 34.0M 4s 2024-03-29T04:10:06.832 INFO:teuthology.orchestra.run.smithi156.stderr: 67850K .......... .......... .......... .......... .......... 40% 32.9M 4s 2024-03-29T04:10:06.833 INFO:teuthology.orchestra.run.smithi156.stderr: 67900K .......... .......... .......... .......... .......... 40% 29.0M 4s 2024-03-29T04:10:06.835 INFO:teuthology.orchestra.run.smithi156.stderr: 67950K .......... .......... .......... .......... .......... 40% 30.8M 4s 2024-03-29T04:10:06.837 INFO:teuthology.orchestra.run.smithi156.stderr: 68000K .......... .......... .......... .......... .......... 40% 27.3M 4s 2024-03-29T04:10:06.839 INFO:teuthology.orchestra.run.smithi156.stderr: 68050K .......... .......... .......... .......... .......... 40% 27.7M 4s 2024-03-29T04:10:06.840 INFO:teuthology.orchestra.run.smithi156.stderr: 68100K .......... .......... .......... .......... .......... 40% 28.7M 4s 2024-03-29T04:10:06.842 INFO:teuthology.orchestra.run.smithi156.stderr: 68150K .......... .......... .......... .......... .......... 40% 32.9M 4s 2024-03-29T04:10:06.843 INFO:teuthology.orchestra.run.smithi156.stderr: 68200K .......... .......... .......... .......... .......... 40% 33.2M 4s 2024-03-29T04:10:06.845 INFO:teuthology.orchestra.run.smithi156.stderr: 68250K .......... .......... .......... .......... .......... 40% 33.3M 4s 2024-03-29T04:10:06.846 INFO:teuthology.orchestra.run.smithi156.stderr: 68300K .......... .......... .......... .......... .......... 40% 25.9M 4s 2024-03-29T04:10:06.848 INFO:teuthology.orchestra.run.smithi156.stderr: 68350K .......... .......... .......... .......... .......... 40% 29.1M 4s 2024-03-29T04:10:06.850 INFO:teuthology.orchestra.run.smithi156.stderr: 68400K .......... .......... .......... .......... .......... 40% 29.6M 4s 2024-03-29T04:10:06.852 INFO:teuthology.orchestra.run.smithi156.stderr: 68450K .......... .......... .......... .......... .......... 40% 28.6M 4s 2024-03-29T04:10:06.854 INFO:teuthology.orchestra.run.smithi156.stderr: 68500K .......... .......... .......... .......... .......... 40% 24.9M 4s 2024-03-29T04:10:06.855 INFO:teuthology.orchestra.run.smithi156.stderr: 68550K .......... .......... .......... .......... .......... 40% 28.2M 4s 2024-03-29T04:10:06.857 INFO:teuthology.orchestra.run.smithi156.stderr: 68600K .......... .......... .......... .......... .......... 40% 26.2M 4s 2024-03-29T04:10:06.859 INFO:teuthology.orchestra.run.smithi156.stderr: 68650K .......... .......... .......... .......... .......... 40% 26.1M 4s 2024-03-29T04:10:06.861 INFO:teuthology.orchestra.run.smithi156.stderr: 68700K .......... .......... .......... .......... .......... 40% 21.8M 4s 2024-03-29T04:10:06.863 INFO:teuthology.orchestra.run.smithi156.stderr: 68750K .......... .......... .......... .......... .......... 40% 23.7M 4s 2024-03-29T04:10:06.865 INFO:teuthology.orchestra.run.smithi156.stderr: 68800K .......... .......... .......... .......... .......... 40% 25.4M 4s 2024-03-29T04:10:06.867 INFO:teuthology.orchestra.run.smithi156.stderr: 68850K .......... .......... .......... .......... .......... 40% 26.6M 4s 2024-03-29T04:10:06.869 INFO:teuthology.orchestra.run.smithi156.stderr: 68900K .......... .......... .......... .......... .......... 40% 23.5M 4s 2024-03-29T04:10:06.871 INFO:teuthology.orchestra.run.smithi156.stderr: 68950K .......... .......... .......... .......... .......... 40% 25.4M 4s 2024-03-29T04:10:06.873 INFO:teuthology.orchestra.run.smithi156.stderr: 69000K .......... .......... .......... .......... .......... 40% 26.8M 4s 2024-03-29T04:10:06.875 INFO:teuthology.orchestra.run.smithi156.stderr: 69050K .......... .......... .......... .......... .......... 40% 27.3M 4s 2024-03-29T04:10:06.877 INFO:teuthology.orchestra.run.smithi156.stderr: 69100K .......... .......... .......... .......... .......... 41% 23.1M 4s 2024-03-29T04:10:06.879 INFO:teuthology.orchestra.run.smithi156.stderr: 69150K .......... .......... .......... .......... .......... 41% 26.3M 4s 2024-03-29T04:10:06.880 INFO:teuthology.orchestra.run.smithi156.stderr: 69200K .......... .......... .......... .......... .......... 41% 26.8M 4s 2024-03-29T04:10:06.883 INFO:teuthology.orchestra.run.smithi156.stderr: 69250K .......... .......... .......... .......... .......... 41% 26.0M 4s 2024-03-29T04:10:06.884 INFO:teuthology.orchestra.run.smithi156.stderr: 69300K .......... .......... .......... .......... .......... 41% 22.3M 4s 2024-03-29T04:10:06.886 INFO:teuthology.orchestra.run.smithi156.stderr: 69350K .......... .......... .......... .......... .......... 41% 26.1M 4s 2024-03-29T04:10:06.888 INFO:teuthology.orchestra.run.smithi156.stderr: 69400K .......... .......... .......... .......... .......... 41% 26.1M 4s 2024-03-29T04:10:06.890 INFO:teuthology.orchestra.run.smithi156.stderr: 69450K .......... .......... .......... .......... .......... 41% 25.4M 4s 2024-03-29T04:10:06.892 INFO:teuthology.orchestra.run.smithi156.stderr: 69500K .......... .......... .......... .......... .......... 41% 22.4M 4s 2024-03-29T04:10:06.894 INFO:teuthology.orchestra.run.smithi156.stderr: 69550K .......... .......... .......... .......... .......... 41% 25.4M 4s 2024-03-29T04:10:06.896 INFO:teuthology.orchestra.run.smithi156.stderr: 69600K .......... .......... .......... .......... .......... 41% 23.2M 4s 2024-03-29T04:10:06.899 INFO:teuthology.orchestra.run.smithi156.stderr: 69650K .......... .......... .......... .......... .......... 41% 23.8M 4s 2024-03-29T04:10:06.901 INFO:teuthology.orchestra.run.smithi156.stderr: 69700K .......... .......... .......... .......... .......... 41% 21.9M 4s 2024-03-29T04:10:06.902 INFO:teuthology.orchestra.run.smithi156.stderr: 69750K .......... .......... .......... .......... .......... 41% 31.7M 4s 2024-03-29T04:10:06.904 INFO:teuthology.orchestra.run.smithi156.stderr: 69800K .......... .......... .......... .......... .......... 41% 33.4M 4s 2024-03-29T04:10:06.905 INFO:teuthology.orchestra.run.smithi156.stderr: 69850K .......... .......... .......... .......... .......... 41% 33.1M 4s 2024-03-29T04:10:06.907 INFO:teuthology.orchestra.run.smithi156.stderr: 69900K .......... .......... .......... .......... .......... 41% 28.8M 4s 2024-03-29T04:10:06.908 INFO:teuthology.orchestra.run.smithi156.stderr: 69950K .......... .......... .......... .......... .......... 41% 38.8M 4s 2024-03-29T04:10:06.909 INFO:teuthology.orchestra.run.smithi156.stderr: 70000K .......... .......... .......... .......... .......... 41% 40.0M 4s 2024-03-29T04:10:06.911 INFO:teuthology.orchestra.run.smithi156.stderr: 70050K .......... .......... .......... .......... .......... 41% 40.7M 4s 2024-03-29T04:10:06.912 INFO:teuthology.orchestra.run.smithi156.stderr: 70100K .......... .......... .......... .......... .......... 41% 35.4M 4s 2024-03-29T04:10:06.913 INFO:teuthology.orchestra.run.smithi156.stderr: 70150K .......... .......... .......... .......... .......... 41% 35.0M 4s 2024-03-29T04:10:06.915 INFO:teuthology.orchestra.run.smithi156.stderr: 70200K .......... .......... .......... .......... .......... 41% 29.1M 4s 2024-03-29T04:10:06.917 INFO:teuthology.orchestra.run.smithi156.stderr: 70250K .......... .......... .......... .......... .......... 41% 32.3M 4s 2024-03-29T04:10:06.919 INFO:teuthology.orchestra.run.smithi156.stderr: 70300K .......... .......... .......... .......... .......... 41% 26.9M 4s 2024-03-29T04:10:06.920 INFO:teuthology.orchestra.run.smithi156.stderr: 70350K .......... .......... .......... .......... .......... 41% 35.5M 4s 2024-03-29T04:10:06.921 INFO:teuthology.orchestra.run.smithi156.stderr: 70400K .......... .......... .......... .......... .......... 41% 37.0M 4s 2024-03-29T04:10:06.922 INFO:teuthology.orchestra.run.smithi156.stderr: 70450K .......... .......... .......... .......... .......... 41% 36.9M 4s 2024-03-29T04:10:06.924 INFO:teuthology.orchestra.run.smithi156.stderr: 70500K .......... .......... .......... .......... .......... 41% 28.7M 4s 2024-03-29T04:10:06.925 INFO:teuthology.orchestra.run.smithi156.stderr: 70550K .......... .......... .......... .......... .......... 41% 37.4M 4s 2024-03-29T04:10:06.926 INFO:teuthology.orchestra.run.smithi156.stderr: 70600K .......... .......... .......... .......... .......... 41% 40.1M 4s 2024-03-29T04:10:06.928 INFO:teuthology.orchestra.run.smithi156.stderr: 70650K .......... .......... .......... .......... .......... 41% 38.1M 4s 2024-03-29T04:10:06.929 INFO:teuthology.orchestra.run.smithi156.stderr: 70700K .......... .......... .......... .......... .......... 41% 33.2M 4s 2024-03-29T04:10:06.931 INFO:teuthology.orchestra.run.smithi156.stderr: 70750K .......... .......... .......... .......... .......... 42% 38.6M 4s 2024-03-29T04:10:06.932 INFO:teuthology.orchestra.run.smithi156.stderr: 70800K .......... .......... .......... .......... .......... 42% 37.8M 4s 2024-03-29T04:10:06.933 INFO:teuthology.orchestra.run.smithi156.stderr: 70850K .......... .......... .......... .......... .......... 42% 38.0M 4s 2024-03-29T04:10:06.935 INFO:teuthology.orchestra.run.smithi156.stderr: 70900K .......... .......... .......... .......... .......... 42% 32.3M 4s 2024-03-29T04:10:06.936 INFO:teuthology.orchestra.run.smithi156.stderr: 70950K .......... .......... .......... .......... .......... 42% 37.9M 4s 2024-03-29T04:10:06.937 INFO:teuthology.orchestra.run.smithi156.stderr: 71000K .......... .......... .......... .......... .......... 42% 37.1M 4s 2024-03-29T04:10:06.939 INFO:teuthology.orchestra.run.smithi156.stderr: 71050K .......... .......... .......... .......... .......... 42% 37.0M 4s 2024-03-29T04:10:06.940 INFO:teuthology.orchestra.run.smithi156.stderr: 71100K .......... .......... .......... .......... .......... 42% 25.8M 4s 2024-03-29T04:10:06.942 INFO:teuthology.orchestra.run.smithi156.stderr: 71150K .......... .......... .......... .......... .......... 42% 30.4M 4s 2024-03-29T04:10:06.944 INFO:teuthology.orchestra.run.smithi156.stderr: 71200K .......... .......... .......... .......... .......... 42% 30.0M 4s 2024-03-29T04:10:06.945 INFO:teuthology.orchestra.run.smithi156.stderr: 71250K .......... .......... .......... .......... .......... 42% 37.4M 4s 2024-03-29T04:10:06.946 INFO:teuthology.orchestra.run.smithi156.stderr: 71300K .......... .......... .......... .......... .......... 42% 31.2M 4s 2024-03-29T04:10:06.948 INFO:teuthology.orchestra.run.smithi156.stderr: 71350K .......... .......... .......... .......... .......... 42% 32.1M 4s 2024-03-29T04:10:06.950 INFO:teuthology.orchestra.run.smithi156.stderr: 71400K .......... .......... .......... .......... .......... 42% 34.6M 4s 2024-03-29T04:10:06.951 INFO:teuthology.orchestra.run.smithi156.stderr: 71450K .......... .......... .......... .......... .......... 42% 34.5M 4s 2024-03-29T04:10:06.953 INFO:teuthology.orchestra.run.smithi156.stderr: 71500K .......... .......... .......... .......... .......... 42% 29.4M 4s 2024-03-29T04:10:06.954 INFO:teuthology.orchestra.run.smithi156.stderr: 71550K .......... .......... .......... .......... .......... 42% 35.6M 4s 2024-03-29T04:10:06.955 INFO:teuthology.orchestra.run.smithi156.stderr: 71600K .......... .......... .......... .......... .......... 42% 35.9M 4s 2024-03-29T04:10:06.957 INFO:teuthology.orchestra.run.smithi156.stderr: 71650K .......... .......... .......... .......... .......... 42% 35.8M 4s 2024-03-29T04:10:06.958 INFO:teuthology.orchestra.run.smithi156.stderr: 71700K .......... .......... .......... .......... .......... 42% 30.0M 4s 2024-03-29T04:10:06.960 INFO:teuthology.orchestra.run.smithi156.stderr: 71750K .......... .......... .......... .......... .......... 42% 35.4M 4s 2024-03-29T04:10:06.961 INFO:teuthology.orchestra.run.smithi156.stderr: 71800K .......... .......... .......... .......... .......... 42% 32.7M 4s 2024-03-29T04:10:06.963 INFO:teuthology.orchestra.run.smithi156.stderr: 71850K .......... .......... .......... .......... .......... 42% 35.5M 4s 2024-03-29T04:10:06.964 INFO:teuthology.orchestra.run.smithi156.stderr: 71900K .......... .......... .......... .......... .......... 42% 30.2M 4s 2024-03-29T04:10:06.966 INFO:teuthology.orchestra.run.smithi156.stderr: 71950K .......... .......... .......... .......... .......... 42% 34.9M 4s 2024-03-29T04:10:06.967 INFO:teuthology.orchestra.run.smithi156.stderr: 72000K .......... .......... .......... .......... .......... 42% 34.3M 4s 2024-03-29T04:10:06.968 INFO:teuthology.orchestra.run.smithi156.stderr: 72050K .......... .......... .......... .......... .......... 42% 34.4M 4s 2024-03-29T04:10:06.970 INFO:teuthology.orchestra.run.smithi156.stderr: 72100K .......... .......... .......... .......... .......... 42% 29.2M 4s 2024-03-29T04:10:06.971 INFO:teuthology.orchestra.run.smithi156.stderr: 72150K .......... .......... .......... .......... .......... 42% 34.3M 4s 2024-03-29T04:10:06.973 INFO:teuthology.orchestra.run.smithi156.stderr: 72200K .......... .......... .......... .......... .......... 42% 33.3M 4s 2024-03-29T04:10:06.975 INFO:teuthology.orchestra.run.smithi156.stderr: 72250K .......... .......... .......... .......... .......... 42% 29.9M 4s 2024-03-29T04:10:06.978 INFO:teuthology.orchestra.run.smithi156.stderr: 72300K .......... .......... .......... .......... .......... 42% 27.8M 4s 2024-03-29T04:10:06.979 INFO:teuthology.orchestra.run.smithi156.stderr: 72350K .......... .......... .......... .......... .......... 42% 34.2M 4s 2024-03-29T04:10:06.980 INFO:teuthology.orchestra.run.smithi156.stderr: 72400K .......... .......... .......... .......... .......... 42% 33.6M 4s 2024-03-29T04:10:06.981 INFO:teuthology.orchestra.run.smithi156.stderr: 72450K .......... .......... .......... .......... .......... 43% 34.6M 4s 2024-03-29T04:10:06.982 INFO:teuthology.orchestra.run.smithi156.stderr: 72500K .......... .......... .......... .......... .......... 43% 28.9M 4s 2024-03-29T04:10:06.984 INFO:teuthology.orchestra.run.smithi156.stderr: 72550K .......... .......... .......... .......... .......... 43% 34.6M 4s 2024-03-29T04:10:06.985 INFO:teuthology.orchestra.run.smithi156.stderr: 72600K .......... .......... .......... .......... .......... 43% 33.6M 4s 2024-03-29T04:10:06.987 INFO:teuthology.orchestra.run.smithi156.stderr: 72650K .......... .......... .......... .......... .......... 43% 34.1M 4s 2024-03-29T04:10:06.988 INFO:teuthology.orchestra.run.smithi156.stderr: 72700K .......... .......... .......... .......... .......... 43% 27.0M 4s 2024-03-29T04:10:06.990 INFO:teuthology.orchestra.run.smithi156.stderr: 72750K .......... .......... .......... .......... .......... 43% 30.9M 4s 2024-03-29T04:10:06.991 INFO:teuthology.orchestra.run.smithi156.stderr: 72800K .......... .......... .......... .......... .......... 43% 30.7M 4s 2024-03-29T04:10:06.994 INFO:teuthology.orchestra.run.smithi156.stderr: 72850K .......... .......... .......... .......... .......... 43% 26.7M 4s 2024-03-29T04:10:06.995 INFO:teuthology.orchestra.run.smithi156.stderr: 72900K .......... .......... .......... .......... .......... 43% 25.8M 4s 2024-03-29T04:10:06.998 INFO:teuthology.orchestra.run.smithi156.stderr: 72950K .......... .......... .......... .......... .......... 43% 30.1M 4s 2024-03-29T04:10:06.999 INFO:teuthology.orchestra.run.smithi156.stderr: 73000K .......... .......... .......... .......... .......... 43% 30.6M 4s 2024-03-29T04:10:07.000 INFO:teuthology.orchestra.run.smithi156.stderr: 73050K .......... .......... .......... .......... .......... 43% 30.7M 4s 2024-03-29T04:10:07.002 INFO:teuthology.orchestra.run.smithi156.stderr: 73100K .......... .......... .......... .......... .......... 43% 26.1M 4s 2024-03-29T04:10:07.004 INFO:teuthology.orchestra.run.smithi156.stderr: 73150K .......... .......... .......... .......... .......... 43% 30.4M 4s 2024-03-29T04:10:07.005 INFO:teuthology.orchestra.run.smithi156.stderr: 73200K .......... .......... .......... .......... .......... 43% 30.1M 4s 2024-03-29T04:10:07.007 INFO:teuthology.orchestra.run.smithi156.stderr: 73250K .......... .......... .......... .......... .......... 43% 30.3M 4s 2024-03-29T04:10:07.009 INFO:teuthology.orchestra.run.smithi156.stderr: 73300K .......... .......... .......... .......... .......... 43% 26.6M 4s 2024-03-29T04:10:07.010 INFO:teuthology.orchestra.run.smithi156.stderr: 73350K .......... .......... .......... .......... .......... 43% 30.9M 4s 2024-03-29T04:10:07.012 INFO:teuthology.orchestra.run.smithi156.stderr: 73400K .......... .......... .......... .......... .......... 43% 30.7M 4s 2024-03-29T04:10:07.014 INFO:teuthology.orchestra.run.smithi156.stderr: 73450K .......... .......... .......... .......... .......... 43% 30.4M 4s 2024-03-29T04:10:07.015 INFO:teuthology.orchestra.run.smithi156.stderr: 73500K .......... .......... .......... .......... .......... 43% 25.9M 4s 2024-03-29T04:10:07.017 INFO:teuthology.orchestra.run.smithi156.stderr: 73550K .......... .......... .......... .......... .......... 43% 30.4M 4s 2024-03-29T04:10:07.019 INFO:teuthology.orchestra.run.smithi156.stderr: 73600K .......... .......... .......... .......... .......... 43% 29.8M 4s 2024-03-29T04:10:07.021 INFO:teuthology.orchestra.run.smithi156.stderr: 73650K .......... .......... .......... .......... .......... 43% 27.8M 4s 2024-03-29T04:10:07.022 INFO:teuthology.orchestra.run.smithi156.stderr: 73700K .......... .......... .......... .......... .......... 43% 24.5M 4s 2024-03-29T04:10:07.024 INFO:teuthology.orchestra.run.smithi156.stderr: 73750K .......... .......... .......... .......... .......... 43% 37.0M 4s 2024-03-29T04:10:07.025 INFO:teuthology.orchestra.run.smithi156.stderr: 73800K .......... .......... .......... .......... .......... 43% 34.3M 4s 2024-03-29T04:10:07.027 INFO:teuthology.orchestra.run.smithi156.stderr: 73850K .......... .......... .......... .......... .......... 43% 37.6M 4s 2024-03-29T04:10:07.028 INFO:teuthology.orchestra.run.smithi156.stderr: 73900K .......... .......... .......... .......... .......... 43% 32.0M 4s 2024-03-29T04:10:07.030 INFO:teuthology.orchestra.run.smithi156.stderr: 73950K .......... .......... .......... .......... .......... 43% 24.3M 4s 2024-03-29T04:10:07.032 INFO:teuthology.orchestra.run.smithi156.stderr: 74000K .......... .......... .......... .......... .......... 43% 28.0M 4s 2024-03-29T04:10:07.033 INFO:teuthology.orchestra.run.smithi156.stderr: 74050K .......... .......... .......... .......... .......... 43% 30.5M 4s 2024-03-29T04:10:07.035 INFO:teuthology.orchestra.run.smithi156.stderr: 74100K .......... .......... .......... .......... .......... 43% 30.5M 4s 2024-03-29T04:10:07.036 INFO:teuthology.orchestra.run.smithi156.stderr: 74150K .......... .......... .......... .......... .......... 44% 31.5M 4s 2024-03-29T04:10:07.038 INFO:teuthology.orchestra.run.smithi156.stderr: 74200K .......... .......... .......... .......... .......... 44% 33.7M 4s 2024-03-29T04:10:07.040 INFO:teuthology.orchestra.run.smithi156.stderr: 74250K .......... .......... .......... .......... .......... 44% 27.0M 4s 2024-03-29T04:10:07.042 INFO:teuthology.orchestra.run.smithi156.stderr: 74300K .......... .......... .......... .......... .......... 44% 25.8M 4s 2024-03-29T04:10:07.043 INFO:teuthology.orchestra.run.smithi156.stderr: 74350K .......... .......... .......... .......... .......... 44% 33.7M 4s 2024-03-29T04:10:07.044 INFO:teuthology.orchestra.run.smithi156.stderr: 74400K .......... .......... .......... .......... .......... 44% 36.0M 4s 2024-03-29T04:10:07.046 INFO:teuthology.orchestra.run.smithi156.stderr: 74450K .......... .......... .......... .......... .......... 44% 38.3M 4s 2024-03-29T04:10:07.047 INFO:teuthology.orchestra.run.smithi156.stderr: 74500K .......... .......... .......... .......... .......... 44% 32.6M 4s 2024-03-29T04:10:07.048 INFO:teuthology.orchestra.run.smithi156.stderr: 74550K .......... .......... .......... .......... .......... 44% 38.4M 4s 2024-03-29T04:10:07.050 INFO:teuthology.orchestra.run.smithi156.stderr: 74600K .......... .......... .......... .......... .......... 44% 36.3M 4s 2024-03-29T04:10:07.051 INFO:teuthology.orchestra.run.smithi156.stderr: 74650K .......... .......... .......... .......... .......... 44% 35.7M 4s 2024-03-29T04:10:07.053 INFO:teuthology.orchestra.run.smithi156.stderr: 74700K .......... .......... .......... .......... .......... 44% 30.5M 4s 2024-03-29T04:10:07.054 INFO:teuthology.orchestra.run.smithi156.stderr: 74750K .......... .......... .......... .......... .......... 44% 33.9M 4s 2024-03-29T04:10:07.056 INFO:teuthology.orchestra.run.smithi156.stderr: 74800K .......... .......... .......... .......... .......... 44% 35.7M 4s 2024-03-29T04:10:07.057 INFO:teuthology.orchestra.run.smithi156.stderr: 74850K .......... .......... .......... .......... .......... 44% 34.6M 4s 2024-03-29T04:10:07.059 INFO:teuthology.orchestra.run.smithi156.stderr: 74900K .......... .......... .......... .......... .......... 44% 31.5M 4s 2024-03-29T04:10:07.060 INFO:teuthology.orchestra.run.smithi156.stderr: 74950K .......... .......... .......... .......... .......... 44% 34.8M 4s 2024-03-29T04:10:07.061 INFO:teuthology.orchestra.run.smithi156.stderr: 75000K .......... .......... .......... .......... .......... 44% 34.9M 4s 2024-03-29T04:10:07.063 INFO:teuthology.orchestra.run.smithi156.stderr: 75050K .......... .......... .......... .......... .......... 44% 34.6M 4s 2024-03-29T04:10:07.064 INFO:teuthology.orchestra.run.smithi156.stderr: 75100K .......... .......... .......... .......... .......... 44% 29.6M 4s 2024-03-29T04:10:07.066 INFO:teuthology.orchestra.run.smithi156.stderr: 75150K .......... .......... .......... .......... .......... 44% 35.7M 4s 2024-03-29T04:10:07.067 INFO:teuthology.orchestra.run.smithi156.stderr: 75200K .......... .......... .......... .......... .......... 44% 34.9M 4s 2024-03-29T04:10:07.069 INFO:teuthology.orchestra.run.smithi156.stderr: 75250K .......... .......... .......... .......... .......... 44% 35.7M 4s 2024-03-29T04:10:07.070 INFO:teuthology.orchestra.run.smithi156.stderr: 75300K .......... .......... .......... .......... .......... 44% 30.3M 4s 2024-03-29T04:10:07.071 INFO:teuthology.orchestra.run.smithi156.stderr: 75350K .......... .......... .......... .......... .......... 44% 35.6M 4s 2024-03-29T04:10:07.073 INFO:teuthology.orchestra.run.smithi156.stderr: 75400K .......... .......... .......... .......... .......... 44% 35.8M 4s 2024-03-29T04:10:07.074 INFO:teuthology.orchestra.run.smithi156.stderr: 75450K .......... .......... .......... .......... .......... 44% 34.9M 4s 2024-03-29T04:10:07.076 INFO:teuthology.orchestra.run.smithi156.stderr: 75500K .......... .......... .......... .......... .......... 44% 30.3M 4s 2024-03-29T04:10:07.077 INFO:teuthology.orchestra.run.smithi156.stderr: 75550K .......... .......... .......... .......... .......... 44% 34.5M 4s 2024-03-29T04:10:07.079 INFO:teuthology.orchestra.run.smithi156.stderr: 75600K .......... .......... .......... .......... .......... 44% 35.4M 4s 2024-03-29T04:10:07.080 INFO:teuthology.orchestra.run.smithi156.stderr: 75650K .......... .......... .......... .......... .......... 44% 34.7M 4s 2024-03-29T04:10:07.082 INFO:teuthology.orchestra.run.smithi156.stderr: 75700K .......... .......... .......... .......... .......... 44% 30.2M 4s 2024-03-29T04:10:07.083 INFO:teuthology.orchestra.run.smithi156.stderr: 75750K .......... .......... .......... .......... .......... 44% 34.3M 4s 2024-03-29T04:10:07.084 INFO:teuthology.orchestra.run.smithi156.stderr: 75800K .......... .......... .......... .......... .......... 44% 34.3M 4s 2024-03-29T04:10:07.086 INFO:teuthology.orchestra.run.smithi156.stderr: 75850K .......... .......... .......... .......... .......... 45% 34.1M 4s 2024-03-29T04:10:07.088 INFO:teuthology.orchestra.run.smithi156.stderr: 75900K .......... .......... .......... .......... .......... 45% 29.5M 4s 2024-03-29T04:10:07.089 INFO:teuthology.orchestra.run.smithi156.stderr: 75950K .......... .......... .......... .......... .......... 45% 34.0M 4s 2024-03-29T04:10:07.090 INFO:teuthology.orchestra.run.smithi156.stderr: 76000K .......... .......... .......... .......... .......... 45% 34.3M 4s 2024-03-29T04:10:07.092 INFO:teuthology.orchestra.run.smithi156.stderr: 76050K .......... .......... .......... .......... .......... 45% 33.8M 4s 2024-03-29T04:10:07.094 INFO:teuthology.orchestra.run.smithi156.stderr: 76100K .......... .......... .......... .......... .......... 45% 28.8M 4s 2024-03-29T04:10:07.095 INFO:teuthology.orchestra.run.smithi156.stderr: 76150K .......... .......... .......... .......... .......... 45% 33.4M 4s 2024-03-29T04:10:07.097 INFO:teuthology.orchestra.run.smithi156.stderr: 76200K .......... .......... .......... .......... .......... 45% 33.5M 4s 2024-03-29T04:10:07.098 INFO:teuthology.orchestra.run.smithi156.stderr: 76250K .......... .......... .......... .......... .......... 45% 33.2M 4s 2024-03-29T04:10:07.100 INFO:teuthology.orchestra.run.smithi156.stderr: 76300K .......... .......... .......... .......... .......... 45% 28.6M 4s 2024-03-29T04:10:07.101 INFO:teuthology.orchestra.run.smithi156.stderr: 76350K .......... .......... .......... .......... .......... 45% 32.7M 4s 2024-03-29T04:10:07.103 INFO:teuthology.orchestra.run.smithi156.stderr: 76400K .......... .......... .......... .......... .......... 45% 32.7M 4s 2024-03-29T04:10:07.104 INFO:teuthology.orchestra.run.smithi156.stderr: 76450K .......... .......... .......... .......... .......... 45% 30.9M 4s 2024-03-29T04:10:07.106 INFO:teuthology.orchestra.run.smithi156.stderr: 76500K .......... .......... .......... .......... .......... 45% 27.8M 4s 2024-03-29T04:10:07.108 INFO:teuthology.orchestra.run.smithi156.stderr: 76550K .......... .......... .......... .......... .......... 45% 29.9M 3s 2024-03-29T04:10:07.109 INFO:teuthology.orchestra.run.smithi156.stderr: 76600K .......... .......... .......... .......... .......... 45% 29.4M 3s 2024-03-29T04:10:07.111 INFO:teuthology.orchestra.run.smithi156.stderr: 76650K .......... .......... .......... .......... .......... 45% 28.6M 3s 2024-03-29T04:10:07.113 INFO:teuthology.orchestra.run.smithi156.stderr: 76700K .......... .......... .......... .......... .......... 45% 25.4M 3s 2024-03-29T04:10:07.114 INFO:teuthology.orchestra.run.smithi156.stderr: 76750K .......... .......... .......... .......... .......... 45% 32.7M 3s 2024-03-29T04:10:07.116 INFO:teuthology.orchestra.run.smithi156.stderr: 76800K .......... .......... .......... .......... .......... 45% 32.8M 3s 2024-03-29T04:10:07.118 INFO:teuthology.orchestra.run.smithi156.stderr: 76850K .......... .......... .......... .......... .......... 45% 33.1M 3s 2024-03-29T04:10:07.120 INFO:teuthology.orchestra.run.smithi156.stderr: 76900K .......... .......... .......... .......... .......... 45% 28.3M 3s 2024-03-29T04:10:07.121 INFO:teuthology.orchestra.run.smithi156.stderr: 76950K .......... .......... .......... .......... .......... 45% 33.7M 3s 2024-03-29T04:10:07.122 INFO:teuthology.orchestra.run.smithi156.stderr: 77000K .......... .......... .......... .......... .......... 45% 31.7M 3s 2024-03-29T04:10:07.124 INFO:teuthology.orchestra.run.smithi156.stderr: 77050K .......... .......... .......... .......... .......... 45% 31.1M 3s 2024-03-29T04:10:07.126 INFO:teuthology.orchestra.run.smithi156.stderr: 77100K .......... .......... .......... .......... .......... 45% 26.2M 3s 2024-03-29T04:10:07.127 INFO:teuthology.orchestra.run.smithi156.stderr: 77150K .......... .......... .......... .......... .......... 45% 28.3M 3s 2024-03-29T04:10:07.129 INFO:teuthology.orchestra.run.smithi156.stderr: 77200K .......... .......... .......... .......... .......... 45% 29.6M 3s 2024-03-29T04:10:07.131 INFO:teuthology.orchestra.run.smithi156.stderr: 77250K .......... .......... .......... .......... .......... 45% 27.6M 3s 2024-03-29T04:10:07.133 INFO:teuthology.orchestra.run.smithi156.stderr: 77300K .......... .......... .......... .......... .......... 45% 25.5M 3s 2024-03-29T04:10:07.134 INFO:teuthology.orchestra.run.smithi156.stderr: 77350K .......... .......... .......... .......... .......... 45% 29.7M 3s 2024-03-29T04:10:07.138 INFO:teuthology.orchestra.run.smithi156.stderr: 77400K .......... .......... .......... .......... .......... 45% 29.4M 3s 2024-03-29T04:10:07.138 INFO:teuthology.orchestra.run.smithi156.stderr: 77450K .......... .......... .......... .......... .......... 45% 30.2M 3s 2024-03-29T04:10:07.140 INFO:teuthology.orchestra.run.smithi156.stderr: 77500K .......... .......... .......... .......... .......... 46% 23.9M 3s 2024-03-29T04:10:07.142 INFO:teuthology.orchestra.run.smithi156.stderr: 77550K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-03-29T04:10:07.143 INFO:teuthology.orchestra.run.smithi156.stderr: 77600K .......... .......... .......... .......... .......... 46% 25.9M 3s 2024-03-29T04:10:07.145 INFO:teuthology.orchestra.run.smithi156.stderr: 77650K .......... .......... .......... .......... .......... 46% 29.8M 3s 2024-03-29T04:10:07.147 INFO:teuthology.orchestra.run.smithi156.stderr: 77700K .......... .......... .......... .......... .......... 46% 25.2M 3s 2024-03-29T04:10:07.149 INFO:teuthology.orchestra.run.smithi156.stderr: 77750K .......... .......... .......... .......... .......... 46% 28.9M 3s 2024-03-29T04:10:07.151 INFO:teuthology.orchestra.run.smithi156.stderr: 77800K .......... .......... .......... .......... .......... 46% 25.9M 3s 2024-03-29T04:10:07.153 INFO:teuthology.orchestra.run.smithi156.stderr: 77850K .......... .......... .......... .......... .......... 46% 26.2M 3s 2024-03-29T04:10:07.155 INFO:teuthology.orchestra.run.smithi156.stderr: 77900K .......... .......... .......... .......... .......... 46% 18.8M 3s 2024-03-29T04:10:07.157 INFO:teuthology.orchestra.run.smithi156.stderr: 77950K .......... .......... .......... .......... .......... 46% 32.1M 3s 2024-03-29T04:10:07.158 INFO:teuthology.orchestra.run.smithi156.stderr: 78000K .......... .......... .......... .......... .......... 46% 36.8M 3s 2024-03-29T04:10:07.159 INFO:teuthology.orchestra.run.smithi156.stderr: 78050K .......... .......... .......... .......... .......... 46% 36.8M 3s 2024-03-29T04:10:07.161 INFO:teuthology.orchestra.run.smithi156.stderr: 78100K .......... .......... .......... .......... .......... 46% 32.7M 3s 2024-03-29T04:10:07.162 INFO:teuthology.orchestra.run.smithi156.stderr: 78150K .......... .......... .......... .......... .......... 46% 36.5M 3s 2024-03-29T04:10:07.163 INFO:teuthology.orchestra.run.smithi156.stderr: 78200K .......... .......... .......... .......... .......... 46% 36.6M 3s 2024-03-29T04:10:07.165 INFO:teuthology.orchestra.run.smithi156.stderr: 78250K .......... .......... .......... .......... .......... 46% 37.6M 3s 2024-03-29T04:10:07.166 INFO:teuthology.orchestra.run.smithi156.stderr: 78300K .......... .......... .......... .......... .......... 46% 30.8M 3s 2024-03-29T04:10:07.168 INFO:teuthology.orchestra.run.smithi156.stderr: 78350K .......... .......... .......... .......... .......... 46% 35.2M 3s 2024-03-29T04:10:07.170 INFO:teuthology.orchestra.run.smithi156.stderr: 78400K .......... .......... .......... .......... .......... 46% 36.8M 3s 2024-03-29T04:10:07.170 INFO:teuthology.orchestra.run.smithi156.stderr: 78450K .......... .......... .......... .......... .......... 46% 37.3M 3s 2024-03-29T04:10:07.173 INFO:teuthology.orchestra.run.smithi156.stderr: 78500K .......... .......... .......... .......... .......... 46% 23.4M 3s 2024-03-29T04:10:07.175 INFO:teuthology.orchestra.run.smithi156.stderr: 78550K .......... .......... .......... .......... .......... 46% 27.7M 3s 2024-03-29T04:10:07.176 INFO:teuthology.orchestra.run.smithi156.stderr: 78600K .......... .......... .......... .......... .......... 46% 38.3M 3s 2024-03-29T04:10:07.177 INFO:teuthology.orchestra.run.smithi156.stderr: 78650K .......... .......... .......... .......... .......... 46% 36.9M 3s 2024-03-29T04:10:07.178 INFO:teuthology.orchestra.run.smithi156.stderr: 78700K .......... .......... .......... .......... .......... 46% 33.5M 3s 2024-03-29T04:10:07.180 INFO:teuthology.orchestra.run.smithi156.stderr: 78750K .......... .......... .......... .......... .......... 46% 25.8M 3s 2024-03-29T04:10:07.182 INFO:teuthology.orchestra.run.smithi156.stderr: 78800K .......... .......... .......... .......... .......... 46% 25.6M 3s 2024-03-29T04:10:07.184 INFO:teuthology.orchestra.run.smithi156.stderr: 78850K .......... .......... .......... .......... .......... 46% 26.7M 3s 2024-03-29T04:10:07.186 INFO:teuthology.orchestra.run.smithi156.stderr: 78900K .......... .......... .......... .......... .......... 46% 23.2M 3s 2024-03-29T04:10:07.188 INFO:teuthology.orchestra.run.smithi156.stderr: 78950K .......... .......... .......... .......... .......... 46% 26.4M 3s 2024-03-29T04:10:07.190 INFO:teuthology.orchestra.run.smithi156.stderr: 79000K .......... .......... .......... .......... .......... 46% 28.4M 3s 2024-03-29T04:10:07.191 INFO:teuthology.orchestra.run.smithi156.stderr: 79050K .......... .......... .......... .......... .......... 46% 31.7M 3s 2024-03-29T04:10:07.193 INFO:teuthology.orchestra.run.smithi156.stderr: 79100K .......... .......... .......... .......... .......... 46% 25.3M 3s 2024-03-29T04:10:07.195 INFO:teuthology.orchestra.run.smithi156.stderr: 79150K .......... .......... .......... .......... .......... 46% 29.5M 3s 2024-03-29T04:10:07.196 INFO:teuthology.orchestra.run.smithi156.stderr: 79200K .......... .......... .......... .......... .......... 47% 29.6M 3s 2024-03-29T04:10:07.198 INFO:teuthology.orchestra.run.smithi156.stderr: 79250K .......... .......... .......... .......... .......... 47% 30.3M 3s 2024-03-29T04:10:07.200 INFO:teuthology.orchestra.run.smithi156.stderr: 79300K .......... .......... .......... .......... .......... 47% 25.2M 3s 2024-03-29T04:10:07.202 INFO:teuthology.orchestra.run.smithi156.stderr: 79350K .......... .......... .......... .......... .......... 47% 27.1M 3s 2024-03-29T04:10:07.204 INFO:teuthology.orchestra.run.smithi156.stderr: 79400K .......... .......... .......... .......... .......... 47% 29.3M 3s 2024-03-29T04:10:07.207 INFO:teuthology.orchestra.run.smithi156.stderr: 79450K .......... .......... .......... .......... .......... 47% 28.0M 3s 2024-03-29T04:10:07.208 INFO:teuthology.orchestra.run.smithi156.stderr: 79500K .......... .......... .......... .......... .......... 47% 24.8M 3s 2024-03-29T04:10:07.209 INFO:teuthology.orchestra.run.smithi156.stderr: 79550K .......... .......... .......... .......... .......... 47% 28.2M 3s 2024-03-29T04:10:07.210 INFO:teuthology.orchestra.run.smithi156.stderr: 79600K .......... .......... .......... .......... .......... 47% 27.5M 3s 2024-03-29T04:10:07.212 INFO:teuthology.orchestra.run.smithi156.stderr: 79650K .......... .......... .......... .......... .......... 47% 29.0M 3s 2024-03-29T04:10:07.214 INFO:teuthology.orchestra.run.smithi156.stderr: 79700K .......... .......... .......... .......... .......... 47% 24.8M 3s 2024-03-29T04:10:07.216 INFO:teuthology.orchestra.run.smithi156.stderr: 79750K .......... .......... .......... .......... .......... 47% 27.9M 3s 2024-03-29T04:10:07.218 INFO:teuthology.orchestra.run.smithi156.stderr: 79800K .......... .......... .......... .......... .......... 47% 26.1M 3s 2024-03-29T04:10:07.220 INFO:teuthology.orchestra.run.smithi156.stderr: 79850K .......... .......... .......... .......... .......... 47% 26.2M 3s 2024-03-29T04:10:07.222 INFO:teuthology.orchestra.run.smithi156.stderr: 79900K .......... .......... .......... .......... .......... 47% 21.8M 3s 2024-03-29T04:10:07.224 INFO:teuthology.orchestra.run.smithi156.stderr: 79950K .......... .......... .......... .......... .......... 47% 25.2M 3s 2024-03-29T04:10:07.226 INFO:teuthology.orchestra.run.smithi156.stderr: 80000K .......... .......... .......... .......... .......... 47% 25.2M 3s 2024-03-29T04:10:07.228 INFO:teuthology.orchestra.run.smithi156.stderr: 80050K .......... .......... .......... .......... .......... 47% 26.2M 3s 2024-03-29T04:10:07.230 INFO:teuthology.orchestra.run.smithi156.stderr: 80100K .......... .......... .......... .......... .......... 47% 22.5M 3s 2024-03-29T04:10:07.232 INFO:teuthology.orchestra.run.smithi156.stderr: 80150K .......... .......... .......... .......... .......... 47% 27.0M 3s 2024-03-29T04:10:07.234 INFO:teuthology.orchestra.run.smithi156.stderr: 80200K .......... .......... .......... .......... .......... 47% 24.2M 3s 2024-03-29T04:10:07.236 INFO:teuthology.orchestra.run.smithi156.stderr: 80250K .......... .......... .......... .......... .......... 47% 28.0M 3s 2024-03-29T04:10:07.237 INFO:teuthology.orchestra.run.smithi156.stderr: 80300K .......... .......... .......... .......... .......... 47% 24.3M 3s 2024-03-29T04:10:07.239 INFO:teuthology.orchestra.run.smithi156.stderr: 80350K .......... .......... .......... .......... .......... 47% 28.2M 3s 2024-03-29T04:10:07.241 INFO:teuthology.orchestra.run.smithi156.stderr: 80400K .......... .......... .......... .......... .......... 47% 28.5M 3s 2024-03-29T04:10:07.243 INFO:teuthology.orchestra.run.smithi156.stderr: 80450K .......... .......... .......... .......... .......... 47% 28.1M 3s 2024-03-29T04:10:07.245 INFO:teuthology.orchestra.run.smithi156.stderr: 80500K .......... .......... .......... .......... .......... 47% 23.5M 3s 2024-03-29T04:10:07.247 INFO:teuthology.orchestra.run.smithi156.stderr: 80550K .......... .......... .......... .......... .......... 47% 28.1M 3s 2024-03-29T04:10:07.249 INFO:teuthology.orchestra.run.smithi156.stderr: 80600K .......... .......... .......... .......... .......... 47% 26.5M 3s 2024-03-29T04:10:07.251 INFO:teuthology.orchestra.run.smithi156.stderr: 80650K .......... .......... .......... .......... .......... 47% 26.9M 3s 2024-03-29T04:10:07.252 INFO:teuthology.orchestra.run.smithi156.stderr: 80700K .......... .......... .......... .......... .......... 47% 24.2M 3s 2024-03-29T04:10:07.254 INFO:teuthology.orchestra.run.smithi156.stderr: 80750K .......... .......... .......... .......... .......... 47% 27.4M 3s 2024-03-29T04:10:07.256 INFO:teuthology.orchestra.run.smithi156.stderr: 80800K .......... .......... .......... .......... .......... 47% 26.2M 3s 2024-03-29T04:10:07.258 INFO:teuthology.orchestra.run.smithi156.stderr: 80850K .......... .......... .......... .......... .......... 47% 27.2M 3s 2024-03-29T04:10:07.260 INFO:teuthology.orchestra.run.smithi156.stderr: 80900K .......... .......... .......... .......... .......... 48% 22.8M 3s 2024-03-29T04:10:07.261 INFO:teuthology.orchestra.run.smithi156.stderr: 80950K .......... .......... .......... .......... .......... 48% 27.0M 3s 2024-03-29T04:10:07.264 INFO:teuthology.orchestra.run.smithi156.stderr: 81000K .......... .......... .......... .......... .......... 48% 26.6M 3s 2024-03-29T04:10:07.265 INFO:teuthology.orchestra.run.smithi156.stderr: 81050K .......... .......... .......... .......... .......... 48% 27.2M 3s 2024-03-29T04:10:07.267 INFO:teuthology.orchestra.run.smithi156.stderr: 81100K .......... .......... .......... .......... .......... 48% 23.3M 3s 2024-03-29T04:10:07.269 INFO:teuthology.orchestra.run.smithi156.stderr: 81150K .......... .......... .......... .......... .......... 48% 27.2M 3s 2024-03-29T04:10:07.271 INFO:teuthology.orchestra.run.smithi156.stderr: 81200K .......... .......... .......... .......... .......... 48% 26.9M 3s 2024-03-29T04:10:07.273 INFO:teuthology.orchestra.run.smithi156.stderr: 81250K .......... .......... .......... .......... .......... 48% 26.4M 3s 2024-03-29T04:10:07.275 INFO:teuthology.orchestra.run.smithi156.stderr: 81300K .......... .......... .......... .......... .......... 48% 22.0M 3s 2024-03-29T04:10:07.277 INFO:teuthology.orchestra.run.smithi156.stderr: 81350K .......... .......... .......... .......... .......... 48% 26.3M 3s 2024-03-29T04:10:07.279 INFO:teuthology.orchestra.run.smithi156.stderr: 81400K .......... .......... .......... .......... .......... 48% 24.8M 3s 2024-03-29T04:10:07.281 INFO:teuthology.orchestra.run.smithi156.stderr: 81450K .......... .......... .......... .......... .......... 48% 25.3M 3s 2024-03-29T04:10:07.283 INFO:teuthology.orchestra.run.smithi156.stderr: 81500K .......... .......... .......... .......... .......... 48% 21.8M 3s 2024-03-29T04:10:07.285 INFO:teuthology.orchestra.run.smithi156.stderr: 81550K .......... .......... .......... .......... .......... 48% 26.2M 3s 2024-03-29T04:10:07.287 INFO:teuthology.orchestra.run.smithi156.stderr: 81600K .......... .......... .......... .......... .......... 48% 26.4M 3s 2024-03-29T04:10:07.289 INFO:teuthology.orchestra.run.smithi156.stderr: 81650K .......... .......... .......... .......... .......... 48% 27.0M 3s 2024-03-29T04:10:07.291 INFO:teuthology.orchestra.run.smithi156.stderr: 81700K .......... .......... .......... .......... .......... 48% 22.1M 3s 2024-03-29T04:10:07.292 INFO:teuthology.orchestra.run.smithi156.stderr: 81750K .......... .......... .......... .......... .......... 48% 27.1M 3s 2024-03-29T04:10:07.294 INFO:teuthology.orchestra.run.smithi156.stderr: 81800K .......... .......... .......... .......... .......... 48% 27.2M 3s 2024-03-29T04:10:07.296 INFO:teuthology.orchestra.run.smithi156.stderr: 81850K .......... .......... .......... .......... .......... 48% 26.9M 3s 2024-03-29T04:10:07.298 INFO:teuthology.orchestra.run.smithi156.stderr: 81900K .......... .......... .......... .......... .......... 48% 22.6M 3s 2024-03-29T04:10:07.300 INFO:teuthology.orchestra.run.smithi156.stderr: 81950K .......... .......... .......... .......... .......... 48% 29.3M 3s 2024-03-29T04:10:07.301 INFO:teuthology.orchestra.run.smithi156.stderr: 82000K .......... .......... .......... .......... .......... 48% 31.7M 3s 2024-03-29T04:10:07.305 INFO:teuthology.orchestra.run.smithi156.stderr: 82050K .......... .......... .......... .......... .......... 48% 25.7M 3s 2024-03-29T04:10:07.306 INFO:teuthology.orchestra.run.smithi156.stderr: 82100K .......... .......... .......... .......... .......... 48% 21.2M 3s 2024-03-29T04:10:07.307 INFO:teuthology.orchestra.run.smithi156.stderr: 82150K .......... .......... .......... .......... .......... 48% 29.6M 3s 2024-03-29T04:10:07.309 INFO:teuthology.orchestra.run.smithi156.stderr: 82200K .......... .......... .......... .......... .......... 48% 30.0M 3s 2024-03-29T04:10:07.311 INFO:teuthology.orchestra.run.smithi156.stderr: 82250K .......... .......... .......... .......... .......... 48% 24.5M 3s 2024-03-29T04:10:07.313 INFO:teuthology.orchestra.run.smithi156.stderr: 82300K .......... .......... .......... .......... .......... 48% 24.5M 3s 2024-03-29T04:10:07.314 INFO:teuthology.orchestra.run.smithi156.stderr: 82350K .......... .......... .......... .......... .......... 48% 29.6M 3s 2024-03-29T04:10:07.316 INFO:teuthology.orchestra.run.smithi156.stderr: 82400K .......... .......... .......... .......... .......... 48% 29.2M 3s 2024-03-29T04:10:07.318 INFO:teuthology.orchestra.run.smithi156.stderr: 82450K .......... .......... .......... .......... .......... 48% 29.6M 3s 2024-03-29T04:10:07.320 INFO:teuthology.orchestra.run.smithi156.stderr: 82500K .......... .......... .......... .......... .......... 48% 25.8M 3s 2024-03-29T04:10:07.321 INFO:teuthology.orchestra.run.smithi156.stderr: 82550K .......... .......... .......... .......... .......... 49% 29.7M 3s 2024-03-29T04:10:07.323 INFO:teuthology.orchestra.run.smithi156.stderr: 82600K .......... .......... .......... .......... .......... 49% 29.4M 3s 2024-03-29T04:10:07.325 INFO:teuthology.orchestra.run.smithi156.stderr: 82650K .......... .......... .......... .......... .......... 49% 29.5M 3s 2024-03-29T04:10:07.327 INFO:teuthology.orchestra.run.smithi156.stderr: 82700K .......... .......... .......... .......... .......... 49% 25.4M 3s 2024-03-29T04:10:07.328 INFO:teuthology.orchestra.run.smithi156.stderr: 82750K .......... .......... .......... .......... .......... 49% 29.1M 3s 2024-03-29T04:10:07.330 INFO:teuthology.orchestra.run.smithi156.stderr: 82800K .......... .......... .......... .......... .......... 49% 28.3M 3s 2024-03-29T04:10:07.333 INFO:teuthology.orchestra.run.smithi156.stderr: 82850K .......... .......... .......... .......... .......... 49% 23.0M 3s 2024-03-29T04:10:07.334 INFO:teuthology.orchestra.run.smithi156.stderr: 82900K .......... .......... .......... .......... .......... 49% 31.2M 3s 2024-03-29T04:10:07.335 INFO:teuthology.orchestra.run.smithi156.stderr: 82950K .......... .......... .......... .......... .......... 49% 36.9M 3s 2024-03-29T04:10:07.336 INFO:teuthology.orchestra.run.smithi156.stderr: 83000K .......... .......... .......... .......... .......... 49% 38.1M 3s 2024-03-29T04:10:07.338 INFO:teuthology.orchestra.run.smithi156.stderr: 83050K .......... .......... .......... .......... .......... 49% 38.4M 3s 2024-03-29T04:10:07.339 INFO:teuthology.orchestra.run.smithi156.stderr: 83100K .......... .......... .......... .......... .......... 49% 32.4M 3s 2024-03-29T04:10:07.342 INFO:teuthology.orchestra.run.smithi156.stderr: 83150K .......... .......... .......... .......... .......... 49% 38.5M 3s 2024-03-29T04:10:07.343 INFO:teuthology.orchestra.run.smithi156.stderr: 83200K .......... .......... .......... .......... .......... 49% 38.3M 3s 2024-03-29T04:10:07.343 INFO:teuthology.orchestra.run.smithi156.stderr: 83250K .......... .......... .......... .......... .......... 49% 36.1M 3s 2024-03-29T04:10:07.345 INFO:teuthology.orchestra.run.smithi156.stderr: 83300K .......... .......... .......... .......... .......... 49% 29.9M 3s 2024-03-29T04:10:07.346 INFO:teuthology.orchestra.run.smithi156.stderr: 83350K .......... .......... .......... .......... .......... 49% 35.9M 3s 2024-03-29T04:10:07.348 INFO:teuthology.orchestra.run.smithi156.stderr: 83400K .......... .......... .......... .......... .......... 49% 35.7M 3s 2024-03-29T04:10:07.349 INFO:teuthology.orchestra.run.smithi156.stderr: 83450K .......... .......... .......... .......... .......... 49% 35.3M 3s 2024-03-29T04:10:07.351 INFO:teuthology.orchestra.run.smithi156.stderr: 83500K .......... .......... .......... .......... .......... 49% 31.4M 3s 2024-03-29T04:10:07.352 INFO:teuthology.orchestra.run.smithi156.stderr: 83550K .......... .......... .......... .......... .......... 49% 32.3M 3s 2024-03-29T04:10:07.353 INFO:teuthology.orchestra.run.smithi156.stderr: 83600K .......... .......... .......... .......... .......... 49% 35.8M 3s 2024-03-29T04:10:07.355 INFO:teuthology.orchestra.run.smithi156.stderr: 83650K .......... .......... .......... .......... .......... 49% 34.2M 3s 2024-03-29T04:10:07.356 INFO:teuthology.orchestra.run.smithi156.stderr: 83700K .......... .......... .......... .......... .......... 49% 29.3M 3s 2024-03-29T04:10:07.358 INFO:teuthology.orchestra.run.smithi156.stderr: 83750K .......... .......... .......... .......... .......... 49% 34.6M 3s 2024-03-29T04:10:07.359 INFO:teuthology.orchestra.run.smithi156.stderr: 83800K .......... .......... .......... .......... .......... 49% 33.8M 3s 2024-03-29T04:10:07.360 INFO:teuthology.orchestra.run.smithi156.stderr: 83850K .......... .......... .......... .......... .......... 49% 35.8M 3s 2024-03-29T04:10:07.362 INFO:teuthology.orchestra.run.smithi156.stderr: 83900K .......... .......... .......... .......... .......... 49% 31.0M 3s 2024-03-29T04:10:07.363 INFO:teuthology.orchestra.run.smithi156.stderr: 83950K .......... .......... .......... .......... .......... 49% 36.1M 3s 2024-03-29T04:10:07.365 INFO:teuthology.orchestra.run.smithi156.stderr: 84000K .......... .......... .......... .......... .......... 49% 32.9M 3s 2024-03-29T04:10:07.366 INFO:teuthology.orchestra.run.smithi156.stderr: 84050K .......... .......... .......... .......... .......... 49% 32.9M 3s 2024-03-29T04:10:07.368 INFO:teuthology.orchestra.run.smithi156.stderr: 84100K .......... .......... .......... .......... .......... 49% 27.9M 3s 2024-03-29T04:10:07.370 INFO:teuthology.orchestra.run.smithi156.stderr: 84150K .......... .......... .......... .......... .......... 49% 30.1M 3s 2024-03-29T04:10:07.371 INFO:teuthology.orchestra.run.smithi156.stderr: 84200K .......... .......... .......... .......... .......... 49% 30.5M 3s 2024-03-29T04:10:07.373 INFO:teuthology.orchestra.run.smithi156.stderr: 84250K .......... .......... .......... .......... .......... 50% 29.9M 3s 2024-03-29T04:10:07.375 INFO:teuthology.orchestra.run.smithi156.stderr: 84300K .......... .......... .......... .......... .......... 50% 25.7M 3s 2024-03-29T04:10:07.376 INFO:teuthology.orchestra.run.smithi156.stderr: 84350K .......... .......... .......... .......... .......... 50% 29.3M 3s 2024-03-29T04:10:07.378 INFO:teuthology.orchestra.run.smithi156.stderr: 84400K .......... .......... .......... .......... .......... 50% 28.8M 3s 2024-03-29T04:10:07.380 INFO:teuthology.orchestra.run.smithi156.stderr: 84450K .......... .......... .......... .......... .......... 50% 28.7M 3s 2024-03-29T04:10:07.382 INFO:teuthology.orchestra.run.smithi156.stderr: 84500K .......... .......... .......... .......... .......... 50% 24.6M 3s 2024-03-29T04:10:07.384 INFO:teuthology.orchestra.run.smithi156.stderr: 84550K .......... .......... .......... .......... .......... 50% 28.9M 3s 2024-03-29T04:10:07.385 INFO:teuthology.orchestra.run.smithi156.stderr: 84600K .......... .......... .......... .......... .......... 50% 28.5M 3s 2024-03-29T04:10:07.387 INFO:teuthology.orchestra.run.smithi156.stderr: 84650K .......... .......... .......... .......... .......... 50% 28.9M 3s 2024-03-29T04:10:07.389 INFO:teuthology.orchestra.run.smithi156.stderr: 84700K .......... .......... .......... .......... .......... 50% 24.3M 3s 2024-03-29T04:10:07.391 INFO:teuthology.orchestra.run.smithi156.stderr: 84750K .......... .......... .......... .......... .......... 50% 28.6M 3s 2024-03-29T04:10:07.392 INFO:teuthology.orchestra.run.smithi156.stderr: 84800K .......... .......... .......... .......... .......... 50% 29.2M 3s 2024-03-29T04:10:07.394 INFO:teuthology.orchestra.run.smithi156.stderr: 84850K .......... .......... .......... .......... .......... 50% 28.7M 3s 2024-03-29T04:10:07.396 INFO:teuthology.orchestra.run.smithi156.stderr: 84900K .......... .......... .......... .......... .......... 50% 24.6M 3s 2024-03-29T04:10:07.398 INFO:teuthology.orchestra.run.smithi156.stderr: 84950K .......... .......... .......... .......... .......... 50% 28.7M 3s 2024-03-29T04:10:07.399 INFO:teuthology.orchestra.run.smithi156.stderr: 85000K .......... .......... .......... .......... .......... 50% 28.4M 3s 2024-03-29T04:10:07.402 INFO:teuthology.orchestra.run.smithi156.stderr: 85050K .......... .......... .......... .......... .......... 50% 25.8M 3s 2024-03-29T04:10:07.404 INFO:teuthology.orchestra.run.smithi156.stderr: 85100K .......... .......... .......... .......... .......... 50% 21.6M 3s 2024-03-29T04:10:07.405 INFO:teuthology.orchestra.run.smithi156.stderr: 85150K .......... .......... .......... .......... .......... 50% 27.6M 3s 2024-03-29T04:10:07.407 INFO:teuthology.orchestra.run.smithi156.stderr: 85200K .......... .......... .......... .......... .......... 50% 27.9M 3s 2024-03-29T04:10:07.409 INFO:teuthology.orchestra.run.smithi156.stderr: 85250K .......... .......... .......... .......... .......... 50% 27.1M 3s 2024-03-29T04:10:07.411 INFO:teuthology.orchestra.run.smithi156.stderr: 85300K .......... .......... .......... .......... .......... 50% 23.5M 3s 2024-03-29T04:10:07.413 INFO:teuthology.orchestra.run.smithi156.stderr: 85350K .......... .......... .......... .......... .......... 50% 27.4M 3s 2024-03-29T04:10:07.415 INFO:teuthology.orchestra.run.smithi156.stderr: 85400K .......... .......... .......... .......... .......... 50% 27.3M 3s 2024-03-29T04:10:07.417 INFO:teuthology.orchestra.run.smithi156.stderr: 85450K .......... .......... .......... .......... .......... 50% 26.7M 3s 2024-03-29T04:10:07.419 INFO:teuthology.orchestra.run.smithi156.stderr: 85500K .......... .......... .......... .......... .......... 50% 23.3M 3s 2024-03-29T04:10:07.420 INFO:teuthology.orchestra.run.smithi156.stderr: 85550K .......... .......... .......... .......... .......... 50% 26.6M 3s 2024-03-29T04:10:07.422 INFO:teuthology.orchestra.run.smithi156.stderr: 85600K .......... .......... .......... .......... .......... 50% 26.5M 3s 2024-03-29T04:10:07.424 INFO:teuthology.orchestra.run.smithi156.stderr: 85650K .......... .......... .......... .......... .......... 50% 26.5M 3s 2024-03-29T04:10:07.426 INFO:teuthology.orchestra.run.smithi156.stderr: 85700K .......... .......... .......... .......... .......... 50% 22.7M 3s 2024-03-29T04:10:07.428 INFO:teuthology.orchestra.run.smithi156.stderr: 85750K .......... .......... .......... .......... .......... 50% 26.4M 3s 2024-03-29T04:10:07.430 INFO:teuthology.orchestra.run.smithi156.stderr: 85800K .......... .......... .......... .......... .......... 50% 25.9M 3s 2024-03-29T04:10:07.432 INFO:teuthology.orchestra.run.smithi156.stderr: 85850K .......... .......... .......... .......... .......... 50% 25.5M 3s 2024-03-29T04:10:07.434 INFO:teuthology.orchestra.run.smithi156.stderr: 85900K .......... .......... .......... .......... .......... 50% 21.7M 3s 2024-03-29T04:10:07.436 INFO:teuthology.orchestra.run.smithi156.stderr: 85950K .......... .......... .......... .......... .......... 51% 25.6M 3s 2024-03-29T04:10:07.438 INFO:teuthology.orchestra.run.smithi156.stderr: 86000K .......... .......... .......... .......... .......... 51% 25.6M 3s 2024-03-29T04:10:07.440 INFO:teuthology.orchestra.run.smithi156.stderr: 86050K .......... .......... .......... .......... .......... 51% 31.7M 3s 2024-03-29T04:10:07.443 INFO:teuthology.orchestra.run.smithi156.stderr: 86100K .......... .......... .......... .......... .......... 51% 21.5M 3s 2024-03-29T04:10:07.443 INFO:teuthology.orchestra.run.smithi156.stderr: 86150K .......... .......... .......... .......... .......... 51% 34.5M 3s 2024-03-29T04:10:07.445 INFO:teuthology.orchestra.run.smithi156.stderr: 86200K .......... .......... .......... .......... .......... 51% 34.8M 3s 2024-03-29T04:10:07.446 INFO:teuthology.orchestra.run.smithi156.stderr: 86250K .......... .......... .......... .......... .......... 51% 33.4M 3s 2024-03-29T04:10:07.448 INFO:teuthology.orchestra.run.smithi156.stderr: 86300K .......... .......... .......... .......... .......... 51% 30.5M 3s 2024-03-29T04:10:07.449 INFO:teuthology.orchestra.run.smithi156.stderr: 86350K .......... .......... .......... .......... .......... 51% 34.4M 3s 2024-03-29T04:10:07.451 INFO:teuthology.orchestra.run.smithi156.stderr: 86400K .......... .......... .......... .......... .......... 51% 34.4M 3s 2024-03-29T04:10:07.452 INFO:teuthology.orchestra.run.smithi156.stderr: 86450K .......... .......... .......... .......... .......... 51% 35.0M 3s 2024-03-29T04:10:07.453 INFO:teuthology.orchestra.run.smithi156.stderr: 86500K .......... .......... .......... .......... .......... 51% 27.8M 3s 2024-03-29T04:10:07.455 INFO:teuthology.orchestra.run.smithi156.stderr: 86550K .......... .......... .......... .......... .......... 51% 34.2M 3s 2024-03-29T04:10:07.456 INFO:teuthology.orchestra.run.smithi156.stderr: 86600K .......... .......... .......... .......... .......... 51% 31.7M 3s 2024-03-29T04:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr: 86650K .......... .......... .......... .......... .......... 51% 33.1M 3s 2024-03-29T04:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr: 86700K .......... .......... .......... .......... .......... 51% 27.0M 3s 2024-03-29T04:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr: 86750K .......... .......... .......... .......... .......... 51% 33.0M 3s 2024-03-29T04:10:07.463 INFO:teuthology.orchestra.run.smithi156.stderr: 86800K .......... .......... .......... .......... .......... 51% 32.6M 3s 2024-03-29T04:10:07.465 INFO:teuthology.orchestra.run.smithi156.stderr: 86850K .......... .......... .......... .......... .......... 51% 33.1M 3s 2024-03-29T04:10:07.466 INFO:teuthology.orchestra.run.smithi156.stderr: 86900K .......... .......... .......... .......... .......... 51% 27.7M 3s 2024-03-29T04:10:07.468 INFO:teuthology.orchestra.run.smithi156.stderr: 86950K .......... .......... .......... .......... .......... 51% 33.5M 3s 2024-03-29T04:10:07.469 INFO:teuthology.orchestra.run.smithi156.stderr: 87000K .......... .......... .......... .......... .......... 51% 32.5M 3s 2024-03-29T04:10:07.471 INFO:teuthology.orchestra.run.smithi156.stderr: 87050K .......... .......... .......... .......... .......... 51% 34.0M 3s 2024-03-29T04:10:07.472 INFO:teuthology.orchestra.run.smithi156.stderr: 87100K .......... .......... .......... .......... .......... 51% 27.1M 3s 2024-03-29T04:10:07.474 INFO:teuthology.orchestra.run.smithi156.stderr: 87150K .......... .......... .......... .......... .......... 51% 33.1M 3s 2024-03-29T04:10:07.475 INFO:teuthology.orchestra.run.smithi156.stderr: 87200K .......... .......... .......... .......... .......... 51% 30.1M 3s 2024-03-29T04:10:07.477 INFO:teuthology.orchestra.run.smithi156.stderr: 87250K .......... .......... .......... .......... .......... 51% 31.2M 3s 2024-03-29T04:10:07.479 INFO:teuthology.orchestra.run.smithi156.stderr: 87300K .......... .......... .......... .......... .......... 51% 25.9M 3s 2024-03-29T04:10:07.480 INFO:teuthology.orchestra.run.smithi156.stderr: 87350K .......... .......... .......... .......... .......... 51% 31.1M 3s 2024-03-29T04:10:07.482 INFO:teuthology.orchestra.run.smithi156.stderr: 87400K .......... .......... .......... .......... .......... 51% 31.1M 3s 2024-03-29T04:10:07.484 INFO:teuthology.orchestra.run.smithi156.stderr: 87450K .......... .......... .......... .......... .......... 51% 30.9M 3s 2024-03-29T04:10:07.485 INFO:teuthology.orchestra.run.smithi156.stderr: 87500K .......... .......... .......... .......... .......... 51% 25.5M 3s 2024-03-29T04:10:07.487 INFO:teuthology.orchestra.run.smithi156.stderr: 87550K .......... .......... .......... .......... .......... 51% 28.7M 3s 2024-03-29T04:10:07.489 INFO:teuthology.orchestra.run.smithi156.stderr: 87600K .......... .......... .......... .......... .......... 51% 28.1M 3s 2024-03-29T04:10:07.491 INFO:teuthology.orchestra.run.smithi156.stderr: 87650K .......... .......... .......... .......... .......... 52% 28.4M 3s 2024-03-29T04:10:07.493 INFO:teuthology.orchestra.run.smithi156.stderr: 87700K .......... .......... .......... .......... .......... 52% 24.2M 3s 2024-03-29T04:10:07.494 INFO:teuthology.orchestra.run.smithi156.stderr: 87750K .......... .......... .......... .......... .......... 52% 28.0M 3s 2024-03-29T04:10:07.496 INFO:teuthology.orchestra.run.smithi156.stderr: 87800K .......... .......... .......... .......... .......... 52% 27.8M 3s 2024-03-29T04:10:07.498 INFO:teuthology.orchestra.run.smithi156.stderr: 87850K .......... .......... .......... .......... .......... 52% 28.3M 3s 2024-03-29T04:10:07.500 INFO:teuthology.orchestra.run.smithi156.stderr: 87900K .......... .......... .......... .......... .......... 52% 23.5M 3s 2024-03-29T04:10:07.502 INFO:teuthology.orchestra.run.smithi156.stderr: 87950K .......... .......... .......... .......... .......... 52% 28.2M 3s 2024-03-29T04:10:07.503 INFO:teuthology.orchestra.run.smithi156.stderr: 88000K .......... .......... .......... .......... .......... 52% 33.6M 3s 2024-03-29T04:10:07.505 INFO:teuthology.orchestra.run.smithi156.stderr: 88050K .......... .......... .......... .......... .......... 52% 33.9M 3s 2024-03-29T04:10:07.507 INFO:teuthology.orchestra.run.smithi156.stderr: 88100K .......... .......... .......... .......... .......... 52% 26.9M 3s 2024-03-29T04:10:07.508 INFO:teuthology.orchestra.run.smithi156.stderr: 88150K .......... .......... .......... .......... .......... 52% 28.5M 3s 2024-03-29T04:10:07.510 INFO:teuthology.orchestra.run.smithi156.stderr: 88200K .......... .......... .......... .......... .......... 52% 29.7M 3s 2024-03-29T04:10:07.512 INFO:teuthology.orchestra.run.smithi156.stderr: 88250K .......... .......... .......... .......... .......... 52% 29.6M 3s 2024-03-29T04:10:07.513 INFO:teuthology.orchestra.run.smithi156.stderr: 88300K .......... .......... .......... .......... .......... 52% 24.7M 3s 2024-03-29T04:10:07.515 INFO:teuthology.orchestra.run.smithi156.stderr: 88350K .......... .......... .......... .......... .......... 52% 28.6M 3s 2024-03-29T04:10:07.517 INFO:teuthology.orchestra.run.smithi156.stderr: 88400K .......... .......... .......... .......... .......... 52% 29.2M 3s 2024-03-29T04:10:07.519 INFO:teuthology.orchestra.run.smithi156.stderr: 88450K .......... .......... .......... .......... .......... 52% 29.8M 3s 2024-03-29T04:10:07.520 INFO:teuthology.orchestra.run.smithi156.stderr: 88500K .......... .......... .......... .......... .......... 52% 23.2M 3s 2024-03-29T04:10:07.522 INFO:teuthology.orchestra.run.smithi156.stderr: 88550K .......... .......... .......... .......... .......... 52% 26.2M 3s 2024-03-29T04:10:07.524 INFO:teuthology.orchestra.run.smithi156.stderr: 88600K .......... .......... .......... .......... .......... 52% 27.3M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88650K .......... .......... .......... .......... .......... 52% 26.7M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88700K .......... .......... .......... .......... .......... 52% 24.1M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88750K .......... .......... .......... .......... .......... 52% 29.5M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88800K .......... .......... .......... .......... .......... 52% 29.3M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88850K .......... .......... .......... .......... .......... 52% 30.4M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88900K .......... .......... .......... .......... .......... 52% 25.3M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 88950K .......... .......... .......... .......... .......... 52% 28.9M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 89000K .......... .......... .......... .......... .......... 52% 28.3M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 89050K .......... .......... .......... .......... .......... 52% 29.4M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 89100K .......... .......... .......... .......... .......... 52% 24.4M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 89150K .......... .......... .......... .......... .......... 52% 28.1M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 89200K .......... .......... .......... .......... .......... 52% 28.8M 3s 2024-03-29T04:10:07.733 INFO:teuthology.orchestra.run.smithi156.stderr: 89250K .......... .......... .......... .......... .......... 52% 29.0M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89300K .......... .......... .......... .......... .......... 53% 24.3M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89350K .......... .......... .......... .......... .......... 53% 28.0M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89400K .......... .......... .......... .......... .......... 53% 29.3M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89450K .......... .......... .......... .......... .......... 53% 28.2M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89500K .......... .......... .......... .......... .......... 53% 24.3M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89550K .......... .......... .......... .......... .......... 53% 29.5M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89600K .......... .......... .......... .......... .......... 53% 29.8M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89650K .......... .......... .......... .......... .......... 53% 30.3M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89700K .......... .......... .......... .......... .......... 53% 19.3M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89750K .......... .......... .......... .......... .......... 53% 21.9M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89800K .......... .......... .......... .......... .......... 53% 22.8M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89850K .......... .......... .......... .......... .......... 53% 21.9M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89900K .......... .......... .......... .......... .......... 53% 18.5M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 89950K .......... .......... .......... .......... .......... 53% 22.4M 3s 2024-03-29T04:10:07.734 INFO:teuthology.orchestra.run.smithi156.stderr: 90000K .......... .......... .......... .......... .......... 53% 21.9M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90050K .......... .......... .......... .......... .......... 53% 21.5M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90100K .......... .......... .......... .......... .......... 53% 19.7M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90150K .......... .......... .......... .......... .......... 53% 26.8M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90200K .......... .......... .......... .......... .......... 53% 29.2M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90250K .......... .......... .......... .......... .......... 53% 33.4M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90300K .......... .......... .......... .......... .......... 53% 28.2M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90350K .......... .......... .......... .......... .......... 53% 30.6M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90400K .......... .......... .......... .......... .......... 53% 33.0M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90450K .......... .......... .......... .......... .......... 53% 30.0M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90500K .......... .......... .......... .......... .......... 53% 23.8M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90550K .......... .......... .......... .......... .......... 53% 27.4M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90600K .......... .......... .......... .......... .......... 53% 28.5M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90650K .......... .......... .......... .......... .......... 53% 28.0M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90700K .......... .......... .......... .......... .......... 53% 25.1M 3s 2024-03-29T04:10:07.735 INFO:teuthology.orchestra.run.smithi156.stderr: 90750K .......... .......... .......... .......... .......... 53% 28.9M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 90800K .......... .......... .......... .......... .......... 53% 29.8M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 90850K .......... .......... .......... .......... .......... 53% 28.8M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 90900K .......... .......... .......... .......... .......... 53% 27.0M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 90950K .......... .......... .......... .......... .......... 53% 33.3M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91000K .......... .......... .......... .......... .......... 54% 33.8M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91050K .......... .......... .......... .......... .......... 54% 33.6M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91100K .......... .......... .......... .......... .......... 54% 29.5M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91150K .......... .......... .......... .......... .......... 54% 33.3M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91200K .......... .......... .......... .......... .......... 54% 34.6M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91250K .......... .......... .......... .......... .......... 54% 33.2M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91300K .......... .......... .......... .......... .......... 54% 28.3M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91350K .......... .......... .......... .......... .......... 54% 33.9M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91400K .......... .......... .......... .......... .......... 54% 33.2M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91450K .......... .......... .......... .......... .......... 54% 31.4M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91500K .......... .......... .......... .......... .......... 54% 27.3M 3s 2024-03-29T04:10:07.736 INFO:teuthology.orchestra.run.smithi156.stderr: 91550K .......... .......... .......... .......... .......... 54% 25.1M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91600K .......... .......... .......... .......... .......... 54% 25.5M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91650K .......... .......... .......... .......... .......... 54% 34.4M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91700K .......... .......... .......... .......... .......... 54% 31.2M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91750K .......... .......... .......... .......... .......... 54% 35.0M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91800K .......... .......... .......... .......... .......... 54% 29.4M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91850K .......... .......... .......... .......... .......... 54% 33.2M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91900K .......... .......... .......... .......... .......... 54% 31.1M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 91950K .......... .......... .......... .......... .......... 54% 34.8M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92000K .......... .......... .......... .......... .......... 54% 35.1M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92050K .......... .......... .......... .......... .......... 54% 36.9M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92100K .......... .......... .......... .......... .......... 54% 31.8M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92150K .......... .......... .......... .......... .......... 54% 36.7M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92200K .......... .......... .......... .......... .......... 54% 36.2M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92250K .......... .......... .......... .......... .......... 54% 37.2M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92300K .......... .......... .......... .......... .......... 54% 31.7M 3s 2024-03-29T04:10:07.737 INFO:teuthology.orchestra.run.smithi156.stderr: 92350K .......... .......... .......... .......... .......... 54% 35.8M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92400K .......... .......... .......... .......... .......... 54% 35.3M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92450K .......... .......... .......... .......... .......... 54% 34.1M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92500K .......... .......... .......... .......... .......... 54% 29.1M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92550K .......... .......... .......... .......... .......... 54% 34.3M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92600K .......... .......... .......... .......... .......... 54% 34.6M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92650K .......... .......... .......... .......... .......... 54% 33.8M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92700K .......... .......... .......... .......... .......... 55% 27.8M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92750K .......... .......... .......... .......... .......... 55% 32.0M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92800K .......... .......... .......... .......... .......... 55% 31.7M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92850K .......... .......... .......... .......... .......... 55% 31.5M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92900K .......... .......... .......... .......... .......... 55% 26.8M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 92950K .......... .......... .......... .......... .......... 55% 28.5M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 93000K .......... .......... .......... .......... .......... 55% 32.7M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 93050K .......... .......... .......... .......... .......... 55% 32.0M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 93100K .......... .......... .......... .......... .......... 55% 27.1M 3s 2024-03-29T04:10:07.738 INFO:teuthology.orchestra.run.smithi156.stderr: 93150K .......... .......... .......... .......... .......... 55% 31.3M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93200K .......... .......... .......... .......... .......... 55% 31.6M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93250K .......... .......... .......... .......... .......... 55% 31.3M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93300K .......... .......... .......... .......... .......... 55% 26.7M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93350K .......... .......... .......... .......... .......... 55% 30.8M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93400K .......... .......... .......... .......... .......... 55% 30.8M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93450K .......... .......... .......... .......... .......... 55% 29.7M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93500K .......... .......... .......... .......... .......... 55% 25.5M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93550K .......... .......... .......... .......... .......... 55% 29.8M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93600K .......... .......... .......... .......... .......... 55% 29.4M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93650K .......... .......... .......... .......... .......... 55% 29.4M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93700K .......... .......... .......... .......... .......... 55% 24.3M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93750K .......... .......... .......... .......... .......... 55% 30.0M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93800K .......... .......... .......... .......... .......... 55% 30.4M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93850K .......... .......... .......... .......... .......... 55% 30.2M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93900K .......... .......... .......... .......... .......... 55% 26.1M 3s 2024-03-29T04:10:07.739 INFO:teuthology.orchestra.run.smithi156.stderr: 93950K .......... .......... .......... .......... .......... 55% 29.9M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94000K .......... .......... .......... .......... .......... 55% 29.6M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94050K .......... .......... .......... .......... .......... 55% 29.8M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94100K .......... .......... .......... .......... .......... 55% 25.2M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94150K .......... .......... .......... .......... .......... 55% 29.8M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94200K .......... .......... .......... .......... .......... 55% 28.4M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94250K .......... .......... .......... .......... .......... 55% 30.7M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94300K .......... .......... .......... .......... .......... 55% 24.1M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94350K .......... .......... .......... .......... .......... 56% 37.3M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94400K .......... .......... .......... .......... .......... 56% 35.1M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94450K .......... .......... .......... .......... .......... 56% 35.5M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94500K .......... .......... .......... .......... .......... 56% 28.1M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94550K .......... .......... .......... .......... .......... 56% 35.2M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94600K .......... .......... .......... .......... .......... 56% 32.1M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94650K .......... .......... .......... .......... .......... 56% 34.6M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94700K .......... .......... .......... .......... .......... 56% 31.4M 3s 2024-03-29T04:10:07.740 INFO:teuthology.orchestra.run.smithi156.stderr: 94750K .......... .......... .......... .......... .......... 56% 32.7M 3s 2024-03-29T04:10:07.741 INFO:teuthology.orchestra.run.smithi156.stderr: 94800K .......... .......... .......... .......... .......... 56% 32.5M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 94850K .......... .......... .......... .......... .......... 56% 32.9M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 94900K .......... .......... .......... .......... .......... 56% 27.7M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 94950K .......... .......... .......... .......... .......... 56% 30.5M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 95000K .......... .......... .......... .......... .......... 56% 30.0M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 95050K .......... .......... .......... .......... .......... 56% 30.5M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 95100K .......... .......... .......... .......... .......... 56% 27.0M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 95150K .......... .......... .......... .......... .......... 56% 31.4M 3s 2024-03-29T04:10:07.746 INFO:teuthology.orchestra.run.smithi156.stderr: 95200K .......... .......... .......... .......... .......... 56% 30.3M 3s 2024-03-29T04:10:07.751 INFO:teuthology.orchestra.run.smithi156.stderr: 95250K .......... .......... .......... .......... .......... 56% 30.9M 3s 2024-03-29T04:10:07.751 INFO:teuthology.orchestra.run.smithi156.stderr: 95300K .......... .......... .......... .......... .......... 56% 26.6M 3s 2024-03-29T04:10:07.751 INFO:teuthology.orchestra.run.smithi156.stderr: 95350K .......... .......... .......... .......... .......... 56% 30.9M 3s 2024-03-29T04:10:07.752 INFO:teuthology.orchestra.run.smithi156.stderr: 95400K .......... .......... .......... .......... .......... 56% 30.3M 3s 2024-03-29T04:10:07.754 INFO:teuthology.orchestra.run.smithi156.stderr: 95450K .......... .......... .......... .......... .......... 56% 30.9M 3s 2024-03-29T04:10:07.755 INFO:teuthology.orchestra.run.smithi156.stderr: 95500K .......... .......... .......... .......... .......... 56% 26.4M 3s 2024-03-29T04:10:07.757 INFO:teuthology.orchestra.run.smithi156.stderr: 95550K .......... .......... .......... .......... .......... 56% 30.4M 3s 2024-03-29T04:10:07.758 INFO:teuthology.orchestra.run.smithi156.stderr: 95600K .......... .......... .......... .......... .......... 56% 29.0M 3s 2024-03-29T04:10:07.761 INFO:teuthology.orchestra.run.smithi156.stderr: 95650K .......... .......... .......... .......... .......... 56% 30.6M 3s 2024-03-29T04:10:07.762 INFO:teuthology.orchestra.run.smithi156.stderr: 95700K .......... .......... .......... .......... .......... 56% 25.3M 3s 2024-03-29T04:10:07.764 INFO:teuthology.orchestra.run.smithi156.stderr: 95750K .......... .......... .......... .......... .......... 56% 31.3M 3s 2024-03-29T04:10:07.765 INFO:teuthology.orchestra.run.smithi156.stderr: 95800K .......... .......... .......... .......... .......... 56% 30.2M 3s 2024-03-29T04:10:07.767 INFO:teuthology.orchestra.run.smithi156.stderr: 95850K .......... .......... .......... .......... .......... 56% 28.9M 3s 2024-03-29T04:10:07.769 INFO:teuthology.orchestra.run.smithi156.stderr: 95900K .......... .......... .......... .......... .......... 56% 26.5M 3s 2024-03-29T04:10:07.770 INFO:teuthology.orchestra.run.smithi156.stderr: 95950K .......... .......... .......... .......... .......... 56% 29.1M 3s 2024-03-29T04:10:07.772 INFO:teuthology.orchestra.run.smithi156.stderr: 96000K .......... .......... .......... .......... .......... 56% 29.8M 3s 2024-03-29T04:10:07.774 INFO:teuthology.orchestra.run.smithi156.stderr: 96050K .......... .......... .......... .......... .......... 57% 30.1M 3s 2024-03-29T04:10:07.776 INFO:teuthology.orchestra.run.smithi156.stderr: 96100K .......... .......... .......... .......... .......... 57% 24.8M 3s 2024-03-29T04:10:07.777 INFO:teuthology.orchestra.run.smithi156.stderr: 96150K .......... .......... .......... .......... .......... 57% 29.6M 3s 2024-03-29T04:10:07.779 INFO:teuthology.orchestra.run.smithi156.stderr: 96200K .......... .......... .......... .......... .......... 57% 28.2M 3s 2024-03-29T04:10:07.781 INFO:teuthology.orchestra.run.smithi156.stderr: 96250K .......... .......... .......... .......... .......... 57% 29.1M 3s 2024-03-29T04:10:07.783 INFO:teuthology.orchestra.run.smithi156.stderr: 96300K .......... .......... .......... .......... .......... 57% 25.4M 3s 2024-03-29T04:10:07.784 INFO:teuthology.orchestra.run.smithi156.stderr: 96350K .......... .......... .......... .......... .......... 57% 28.4M 3s 2024-03-29T04:10:07.786 INFO:teuthology.orchestra.run.smithi156.stderr: 96400K .......... .......... .......... .......... .......... 57% 29.0M 3s 2024-03-29T04:10:07.788 INFO:teuthology.orchestra.run.smithi156.stderr: 96450K .......... .......... .......... .......... .......... 57% 29.0M 3s 2024-03-29T04:10:07.790 INFO:teuthology.orchestra.run.smithi156.stderr: 96500K .......... .......... .......... .......... .......... 57% 24.1M 3s 2024-03-29T04:10:07.791 INFO:teuthology.orchestra.run.smithi156.stderr: 96550K .......... .......... .......... .......... .......... 57% 29.1M 3s 2024-03-29T04:10:07.793 INFO:teuthology.orchestra.run.smithi156.stderr: 96600K .......... .......... .......... .......... .......... 57% 28.9M 3s 2024-03-29T04:10:07.795 INFO:teuthology.orchestra.run.smithi156.stderr: 96650K .......... .......... .......... .......... .......... 57% 27.7M 3s 2024-03-29T04:10:07.797 INFO:teuthology.orchestra.run.smithi156.stderr: 96700K .......... .......... .......... .......... .......... 57% 24.7M 3s 2024-03-29T04:10:07.799 INFO:teuthology.orchestra.run.smithi156.stderr: 96750K .......... .......... .......... .......... .......... 57% 27.9M 3s 2024-03-29T04:10:07.800 INFO:teuthology.orchestra.run.smithi156.stderr: 96800K .......... .......... .......... .......... .......... 57% 28.2M 3s 2024-03-29T04:10:07.802 INFO:teuthology.orchestra.run.smithi156.stderr: 96850K .......... .......... .......... .......... .......... 57% 27.4M 3s 2024-03-29T04:10:07.804 INFO:teuthology.orchestra.run.smithi156.stderr: 96900K .......... .......... .......... .......... .......... 57% 24.4M 3s 2024-03-29T04:10:07.806 INFO:teuthology.orchestra.run.smithi156.stderr: 96950K .......... .......... .......... .......... .......... 57% 27.4M 3s 2024-03-29T04:10:07.808 INFO:teuthology.orchestra.run.smithi156.stderr: 97000K .......... .......... .......... .......... .......... 57% 27.4M 3s 2024-03-29T04:10:07.810 INFO:teuthology.orchestra.run.smithi156.stderr: 97050K .......... .......... .......... .......... .......... 57% 27.0M 3s 2024-03-29T04:10:07.812 INFO:teuthology.orchestra.run.smithi156.stderr: 97100K .......... .......... .......... .......... .......... 57% 22.6M 3s 2024-03-29T04:10:07.814 INFO:teuthology.orchestra.run.smithi156.stderr: 97150K .......... .......... .......... .......... .......... 57% 26.6M 3s 2024-03-29T04:10:07.815 INFO:teuthology.orchestra.run.smithi156.stderr: 97200K .......... .......... .......... .......... .......... 57% 25.5M 3s 2024-03-29T04:10:07.818 INFO:teuthology.orchestra.run.smithi156.stderr: 97250K .......... .......... .......... .......... .......... 57% 26.1M 3s 2024-03-29T04:10:07.820 INFO:teuthology.orchestra.run.smithi156.stderr: 97300K .......... .......... .......... .......... .......... 57% 22.2M 3s 2024-03-29T04:10:07.822 INFO:teuthology.orchestra.run.smithi156.stderr: 97350K .......... .......... .......... .......... .......... 57% 24.9M 3s 2024-03-29T04:10:07.823 INFO:teuthology.orchestra.run.smithi156.stderr: 97400K .......... .......... .......... .......... .......... 57% 25.6M 3s 2024-03-29T04:10:07.826 INFO:teuthology.orchestra.run.smithi156.stderr: 97450K .......... .......... .......... .......... .......... 57% 24.5M 3s 2024-03-29T04:10:07.829 INFO:teuthology.orchestra.run.smithi156.stderr: 97500K .......... .......... .......... .......... .......... 57% 21.9M 3s 2024-03-29T04:10:07.831 INFO:teuthology.orchestra.run.smithi156.stderr: 97550K .......... .......... .......... .......... .......... 57% 25.1M 3s 2024-03-29T04:10:07.832 INFO:teuthology.orchestra.run.smithi156.stderr: 97600K .......... .......... .......... .......... .......... 57% 25.1M 3s 2024-03-29T04:10:07.835 INFO:teuthology.orchestra.run.smithi156.stderr: 97650K .......... .......... .......... .......... .......... 57% 24.9M 3s 2024-03-29T04:10:07.837 INFO:teuthology.orchestra.run.smithi156.stderr: 97700K .......... .......... .......... .......... .......... 57% 20.8M 3s 2024-03-29T04:10:07.839 INFO:teuthology.orchestra.run.smithi156.stderr: 97750K .......... .......... .......... .......... .......... 58% 24.4M 3s 2024-03-29T04:10:07.841 INFO:teuthology.orchestra.run.smithi156.stderr: 97800K .......... .......... .......... .......... .......... 58% 24.6M 3s 2024-03-29T04:10:07.843 INFO:teuthology.orchestra.run.smithi156.stderr: 97850K .......... .......... .......... .......... .......... 58% 24.5M 3s 2024-03-29T04:10:07.845 INFO:teuthology.orchestra.run.smithi156.stderr: 97900K .......... .......... .......... .......... .......... 58% 21.6M 3s 2024-03-29T04:10:07.847 INFO:teuthology.orchestra.run.smithi156.stderr: 97950K .......... .......... .......... .......... .......... 58% 26.2M 3s 2024-03-29T04:10:07.848 INFO:teuthology.orchestra.run.smithi156.stderr: 98000K .......... .......... .......... .......... .......... 58% 23.1M 3s 2024-03-29T04:10:07.851 INFO:teuthology.orchestra.run.smithi156.stderr: 98050K .......... .......... .......... .......... .......... 58% 22.2M 3s 2024-03-29T04:10:07.853 INFO:teuthology.orchestra.run.smithi156.stderr: 98100K .......... .......... .......... .......... .......... 58% 19.4M 3s 2024-03-29T04:10:07.855 INFO:teuthology.orchestra.run.smithi156.stderr: 98150K .......... .......... .......... .......... .......... 58% 24.9M 3s 2024-03-29T04:10:07.857 INFO:teuthology.orchestra.run.smithi156.stderr: 98200K .......... .......... .......... .......... .......... 58% 23.9M 3s 2024-03-29T04:10:07.859 INFO:teuthology.orchestra.run.smithi156.stderr: 98250K .......... .......... .......... .......... .......... 58% 25.3M 3s 2024-03-29T04:10:07.862 INFO:teuthology.orchestra.run.smithi156.stderr: 98300K .......... .......... .......... .......... .......... 58% 19.7M 3s 2024-03-29T04:10:07.863 INFO:teuthology.orchestra.run.smithi156.stderr: 98350K .......... .......... .......... .......... .......... 58% 24.8M 3s 2024-03-29T04:10:07.865 INFO:teuthology.orchestra.run.smithi156.stderr: 98400K .......... .......... .......... .......... .......... 58% 21.1M 3s 2024-03-29T04:10:07.867 INFO:teuthology.orchestra.run.smithi156.stderr: 98450K .......... .......... .......... .......... .......... 58% 33.8M 3s 2024-03-29T04:10:07.869 INFO:teuthology.orchestra.run.smithi156.stderr: 98500K .......... .......... .......... .......... .......... 58% 28.1M 3s 2024-03-29T04:10:07.871 INFO:teuthology.orchestra.run.smithi156.stderr: 98550K .......... .......... .......... .......... .......... 58% 33.9M 3s 2024-03-29T04:10:07.872 INFO:teuthology.orchestra.run.smithi156.stderr: 98600K .......... .......... .......... .......... .......... 58% 34.5M 3s 2024-03-29T04:10:07.873 INFO:teuthology.orchestra.run.smithi156.stderr: 98650K .......... .......... .......... .......... .......... 58% 34.2M 3s 2024-03-29T04:10:07.875 INFO:teuthology.orchestra.run.smithi156.stderr: 98700K .......... .......... .......... .......... .......... 58% 29.9M 3s 2024-03-29T04:10:07.876 INFO:teuthology.orchestra.run.smithi156.stderr: 98750K .......... .......... .......... .......... .......... 58% 32.2M 3s 2024-03-29T04:10:07.878 INFO:teuthology.orchestra.run.smithi156.stderr: 98800K .......... .......... .......... .......... .......... 58% 35.2M 3s 2024-03-29T04:10:07.880 INFO:teuthology.orchestra.run.smithi156.stderr: 98850K .......... .......... .......... .......... .......... 58% 31.0M 3s 2024-03-29T04:10:07.881 INFO:teuthology.orchestra.run.smithi156.stderr: 98900K .......... .......... .......... .......... .......... 58% 29.0M 3s 2024-03-29T04:10:07.882 INFO:teuthology.orchestra.run.smithi156.stderr: 98950K .......... .......... .......... .......... .......... 58% 32.0M 3s 2024-03-29T04:10:07.884 INFO:teuthology.orchestra.run.smithi156.stderr: 99000K .......... .......... .......... .......... .......... 58% 30.6M 3s 2024-03-29T04:10:07.886 INFO:teuthology.orchestra.run.smithi156.stderr: 99050K .......... .......... .......... .......... .......... 58% 31.3M 3s 2024-03-29T04:10:07.887 INFO:teuthology.orchestra.run.smithi156.stderr: 99100K .......... .......... .......... .......... .......... 58% 28.2M 3s 2024-03-29T04:10:07.889 INFO:teuthology.orchestra.run.smithi156.stderr: 99150K .......... .......... .......... .......... .......... 58% 31.6M 3s 2024-03-29T04:10:07.891 INFO:teuthology.orchestra.run.smithi156.stderr: 99200K .......... .......... .......... .......... .......... 58% 33.5M 3s 2024-03-29T04:10:07.892 INFO:teuthology.orchestra.run.smithi156.stderr: 99250K .......... .......... .......... .......... .......... 58% 32.6M 3s 2024-03-29T04:10:07.893 INFO:teuthology.orchestra.run.smithi156.stderr: 99300K .......... .......... .......... .......... .......... 58% 27.6M 3s 2024-03-29T04:10:07.895 INFO:teuthology.orchestra.run.smithi156.stderr: 99350K .......... .......... .......... .......... .......... 58% 33.2M 3s 2024-03-29T04:10:07.896 INFO:teuthology.orchestra.run.smithi156.stderr: 99400K .......... .......... .......... .......... .......... 58% 31.3M 3s 2024-03-29T04:10:07.898 INFO:teuthology.orchestra.run.smithi156.stderr: 99450K .......... .......... .......... .......... .......... 59% 31.3M 3s 2024-03-29T04:10:07.900 INFO:teuthology.orchestra.run.smithi156.stderr: 99500K .......... .......... .......... .......... .......... 59% 27.5M 3s 2024-03-29T04:10:07.902 INFO:teuthology.orchestra.run.smithi156.stderr: 99550K .......... .......... .......... .......... .......... 59% 31.4M 3s 2024-03-29T04:10:07.903 INFO:teuthology.orchestra.run.smithi156.stderr: 99600K .......... .......... .......... .......... .......... 59% 31.9M 3s 2024-03-29T04:10:07.905 INFO:teuthology.orchestra.run.smithi156.stderr: 99650K .......... .......... .......... .......... .......... 59% 31.2M 3s 2024-03-29T04:10:07.907 INFO:teuthology.orchestra.run.smithi156.stderr: 99700K .......... .......... .......... .......... .......... 59% 27.1M 3s 2024-03-29T04:10:07.908 INFO:teuthology.orchestra.run.smithi156.stderr: 99750K .......... .......... .......... .......... .......... 59% 31.6M 3s 2024-03-29T04:10:07.910 INFO:teuthology.orchestra.run.smithi156.stderr: 99800K .......... .......... .......... .......... .......... 59% 30.5M 3s 2024-03-29T04:10:07.911 INFO:teuthology.orchestra.run.smithi156.stderr: 99850K .......... .......... .......... .......... .......... 59% 29.5M 3s 2024-03-29T04:10:07.913 INFO:teuthology.orchestra.run.smithi156.stderr: 99900K .......... .......... .......... .......... .......... 59% 26.6M 3s 2024-03-29T04:10:07.914 INFO:teuthology.orchestra.run.smithi156.stderr: 99950K .......... .......... .......... .......... .......... 59% 29.7M 3s 2024-03-29T04:10:07.917 INFO:teuthology.orchestra.run.smithi156.stderr:100000K .......... .......... .......... .......... .......... 59% 28.7M 3s 2024-03-29T04:10:07.918 INFO:teuthology.orchestra.run.smithi156.stderr:100050K .......... .......... .......... .......... .......... 59% 33.2M 3s 2024-03-29T04:10:07.920 INFO:teuthology.orchestra.run.smithi156.stderr:100100K .......... .......... .......... .......... .......... 59% 29.3M 3s 2024-03-29T04:10:07.922 INFO:teuthology.orchestra.run.smithi156.stderr:100150K .......... .......... .......... .......... .......... 59% 34.8M 3s 2024-03-29T04:10:07.922 INFO:teuthology.orchestra.run.smithi156.stderr:100200K .......... .......... .......... .......... .......... 59% 33.9M 3s 2024-03-29T04:10:07.924 INFO:teuthology.orchestra.run.smithi156.stderr:100250K .......... .......... .......... .......... .......... 59% 34.6M 3s 2024-03-29T04:10:07.925 INFO:teuthology.orchestra.run.smithi156.stderr:100300K .......... .......... .......... .......... .......... 59% 29.1M 3s 2024-03-29T04:10:07.927 INFO:teuthology.orchestra.run.smithi156.stderr:100350K .......... .......... .......... .......... .......... 59% 33.7M 3s 2024-03-29T04:10:07.928 INFO:teuthology.orchestra.run.smithi156.stderr:100400K .......... .......... .......... .......... .......... 59% 34.0M 3s 2024-03-29T04:10:07.930 INFO:teuthology.orchestra.run.smithi156.stderr:100450K .......... .......... .......... .......... .......... 59% 33.6M 3s 2024-03-29T04:10:07.931 INFO:teuthology.orchestra.run.smithi156.stderr:100500K .......... .......... .......... .......... .......... 59% 27.3M 3s 2024-03-29T04:10:07.933 INFO:teuthology.orchestra.run.smithi156.stderr:100550K .......... .......... .......... .......... .......... 59% 30.8M 3s 2024-03-29T04:10:07.935 INFO:teuthology.orchestra.run.smithi156.stderr:100600K .......... .......... .......... .......... .......... 59% 31.2M 3s 2024-03-29T04:10:07.936 INFO:teuthology.orchestra.run.smithi156.stderr:100650K .......... .......... .......... .......... .......... 59% 30.6M 3s 2024-03-29T04:10:07.938 INFO:teuthology.orchestra.run.smithi156.stderr:100700K .......... .......... .......... .......... .......... 59% 26.3M 3s 2024-03-29T04:10:07.940 INFO:teuthology.orchestra.run.smithi156.stderr:100750K .......... .......... .......... .......... .......... 59% 30.9M 3s 2024-03-29T04:10:07.941 INFO:teuthology.orchestra.run.smithi156.stderr:100800K .......... .......... .......... .......... .......... 59% 31.7M 3s 2024-03-29T04:10:07.943 INFO:teuthology.orchestra.run.smithi156.stderr:100850K .......... .......... .......... .......... .......... 59% 32.0M 3s 2024-03-29T04:10:07.945 INFO:teuthology.orchestra.run.smithi156.stderr:100900K .......... .......... .......... .......... .......... 59% 26.8M 3s 2024-03-29T04:10:07.946 INFO:teuthology.orchestra.run.smithi156.stderr:100950K .......... .......... .......... .......... .......... 59% 31.8M 3s 2024-03-29T04:10:07.948 INFO:teuthology.orchestra.run.smithi156.stderr:101000K .......... .......... .......... .......... .......... 59% 31.6M 3s 2024-03-29T04:10:07.949 INFO:teuthology.orchestra.run.smithi156.stderr:101050K .......... .......... .......... .......... .......... 59% 31.6M 3s 2024-03-29T04:10:07.955 INFO:teuthology.orchestra.run.smithi156.stderr:101100K .......... .......... .......... .......... .......... 60% 27.0M 3s 2024-03-29T04:10:07.955 INFO:teuthology.orchestra.run.smithi156.stderr:101150K .......... .......... .......... .......... .......... 60% 31.2M 3s 2024-03-29T04:10:07.955 INFO:teuthology.orchestra.run.smithi156.stderr:101200K .......... .......... .......... .......... .......... 60% 31.4M 3s 2024-03-29T04:10:07.956 INFO:teuthology.orchestra.run.smithi156.stderr:101250K .......... .......... .......... .......... .......... 60% 31.0M 2s 2024-03-29T04:10:07.957 INFO:teuthology.orchestra.run.smithi156.stderr:101300K .......... .......... .......... .......... .......... 60% 26.3M 2s 2024-03-29T04:10:07.959 INFO:teuthology.orchestra.run.smithi156.stderr:101350K .......... .......... .......... .......... .......... 60% 30.5M 2s 2024-03-29T04:10:07.961 INFO:teuthology.orchestra.run.smithi156.stderr:101400K .......... .......... .......... .......... .......... 60% 30.8M 2s 2024-03-29T04:10:07.962 INFO:teuthology.orchestra.run.smithi156.stderr:101450K .......... .......... .......... .......... .......... 60% 30.1M 2s 2024-03-29T04:10:07.964 INFO:teuthology.orchestra.run.smithi156.stderr:101500K .......... .......... .......... .......... .......... 60% 25.7M 2s 2024-03-29T04:10:07.966 INFO:teuthology.orchestra.run.smithi156.stderr:101550K .......... .......... .......... .......... .......... 60% 30.2M 2s 2024-03-29T04:10:07.967 INFO:teuthology.orchestra.run.smithi156.stderr:101600K .......... .......... .......... .......... .......... 60% 30.1M 2s 2024-03-29T04:10:07.969 INFO:teuthology.orchestra.run.smithi156.stderr:101650K .......... .......... .......... .......... .......... 60% 30.1M 2s 2024-03-29T04:10:07.971 INFO:teuthology.orchestra.run.smithi156.stderr:101700K .......... .......... .......... .......... .......... 60% 25.3M 2s 2024-03-29T04:10:07.973 INFO:teuthology.orchestra.run.smithi156.stderr:101750K .......... .......... .......... .......... .......... 60% 29.6M 2s 2024-03-29T04:10:07.974 INFO:teuthology.orchestra.run.smithi156.stderr:101800K .......... .......... .......... .......... .......... 60% 28.1M 2s 2024-03-29T04:10:07.976 INFO:teuthology.orchestra.run.smithi156.stderr:101850K .......... .......... .......... .......... .......... 60% 28.4M 2s 2024-03-29T04:10:07.978 INFO:teuthology.orchestra.run.smithi156.stderr:101900K .......... .......... .......... .......... .......... 60% 24.1M 2s 2024-03-29T04:10:07.980 INFO:teuthology.orchestra.run.smithi156.stderr:101950K .......... .......... .......... .......... .......... 60% 27.7M 2s 2024-03-29T04:10:07.982 INFO:teuthology.orchestra.run.smithi156.stderr:102000K .......... .......... .......... .......... .......... 60% 29.2M 2s 2024-03-29T04:10:07.983 INFO:teuthology.orchestra.run.smithi156.stderr:102050K .......... .......... .......... .......... .......... 60% 29.3M 2s 2024-03-29T04:10:07.985 INFO:teuthology.orchestra.run.smithi156.stderr:102100K .......... .......... .......... .......... .......... 60% 25.0M 2s 2024-03-29T04:10:07.987 INFO:teuthology.orchestra.run.smithi156.stderr:102150K .......... .......... .......... .......... .......... 60% 29.0M 2s 2024-03-29T04:10:07.988 INFO:teuthology.orchestra.run.smithi156.stderr:102200K .......... .......... .......... .......... .......... 60% 29.1M 2s 2024-03-29T04:10:07.990 INFO:teuthology.orchestra.run.smithi156.stderr:102250K .......... .......... .......... .......... .......... 60% 29.1M 2s 2024-03-29T04:10:07.992 INFO:teuthology.orchestra.run.smithi156.stderr:102300K .......... .......... .......... .......... .......... 60% 24.4M 2s 2024-03-29T04:10:07.994 INFO:teuthology.orchestra.run.smithi156.stderr:102350K .......... .......... .......... .......... .......... 60% 28.7M 2s 2024-03-29T04:10:07.996 INFO:teuthology.orchestra.run.smithi156.stderr:102400K .......... .......... .......... .......... .......... 60% 23.6M 2s 2024-03-29T04:10:07.998 INFO:teuthology.orchestra.run.smithi156.stderr:102450K .......... .......... .......... .......... .......... 60% 26.5M 2s 2024-03-29T04:10:08.001 INFO:teuthology.orchestra.run.smithi156.stderr:102500K .......... .......... .......... .......... .......... 60% 18.3M 2s 2024-03-29T04:10:08.002 INFO:teuthology.orchestra.run.smithi156.stderr:102550K .......... .......... .......... .......... .......... 60% 27.0M 2s 2024-03-29T04:10:08.004 INFO:teuthology.orchestra.run.smithi156.stderr:102600K .......... .......... .......... .......... .......... 60% 27.5M 2s 2024-03-29T04:10:08.006 INFO:teuthology.orchestra.run.smithi156.stderr:102650K .......... .......... .......... .......... .......... 60% 26.8M 2s 2024-03-29T04:10:08.008 INFO:teuthology.orchestra.run.smithi156.stderr:102700K .......... .......... .......... .......... .......... 60% 23.3M 2s 2024-03-29T04:10:08.010 INFO:teuthology.orchestra.run.smithi156.stderr:102750K .......... .......... .......... .......... .......... 60% 28.3M 2s 2024-03-29T04:10:08.011 INFO:teuthology.orchestra.run.smithi156.stderr:102800K .......... .......... .......... .......... .......... 61% 30.2M 2s 2024-03-29T04:10:08.013 INFO:teuthology.orchestra.run.smithi156.stderr:102850K .......... .......... .......... .......... .......... 61% 30.3M 2s 2024-03-29T04:10:08.015 INFO:teuthology.orchestra.run.smithi156.stderr:102900K .......... .......... .......... .......... .......... 61% 27.8M 2s 2024-03-29T04:10:08.016 INFO:teuthology.orchestra.run.smithi156.stderr:102950K .......... .......... .......... .......... .......... 61% 32.7M 2s 2024-03-29T04:10:08.018 INFO:teuthology.orchestra.run.smithi156.stderr:103000K .......... .......... .......... .......... .......... 61% 33.4M 2s 2024-03-29T04:10:08.019 INFO:teuthology.orchestra.run.smithi156.stderr:103050K .......... .......... .......... .......... .......... 61% 33.1M 2s 2024-03-29T04:10:08.021 INFO:teuthology.orchestra.run.smithi156.stderr:103100K .......... .......... .......... .......... .......... 61% 24.1M 2s 2024-03-29T04:10:08.023 INFO:teuthology.orchestra.run.smithi156.stderr:103150K .......... .......... .......... .......... .......... 61% 28.3M 2s 2024-03-29T04:10:08.024 INFO:teuthology.orchestra.run.smithi156.stderr:103200K .......... .......... .......... .......... .......... 61% 36.3M 2s 2024-03-29T04:10:08.026 INFO:teuthology.orchestra.run.smithi156.stderr:103250K .......... .......... .......... .......... .......... 61% 34.9M 2s 2024-03-29T04:10:08.027 INFO:teuthology.orchestra.run.smithi156.stderr:103300K .......... .......... .......... .......... .......... 61% 31.7M 2s 2024-03-29T04:10:08.029 INFO:teuthology.orchestra.run.smithi156.stderr:103350K .......... .......... .......... .......... .......... 61% 36.1M 2s 2024-03-29T04:10:08.030 INFO:teuthology.orchestra.run.smithi156.stderr:103400K .......... .......... .......... .......... .......... 61% 37.2M 2s 2024-03-29T04:10:08.031 INFO:teuthology.orchestra.run.smithi156.stderr:103450K .......... .......... .......... .......... .......... 61% 36.3M 2s 2024-03-29T04:10:08.033 INFO:teuthology.orchestra.run.smithi156.stderr:103500K .......... .......... .......... .......... .......... 61% 32.5M 2s 2024-03-29T04:10:08.034 INFO:teuthology.orchestra.run.smithi156.stderr:103550K .......... .......... .......... .......... .......... 61% 33.8M 2s 2024-03-29T04:10:08.036 INFO:teuthology.orchestra.run.smithi156.stderr:103600K .......... .......... .......... .......... .......... 61% 35.7M 2s 2024-03-29T04:10:08.037 INFO:teuthology.orchestra.run.smithi156.stderr:103650K .......... .......... .......... .......... .......... 61% 34.4M 2s 2024-03-29T04:10:08.039 INFO:teuthology.orchestra.run.smithi156.stderr:103700K .......... .......... .......... .......... .......... 61% 30.8M 2s 2024-03-29T04:10:08.040 INFO:teuthology.orchestra.run.smithi156.stderr:103750K .......... .......... .......... .......... .......... 61% 35.3M 2s 2024-03-29T04:10:08.042 INFO:teuthology.orchestra.run.smithi156.stderr:103800K .......... .......... .......... .......... .......... 61% 25.8M 2s 2024-03-29T04:10:08.044 INFO:teuthology.orchestra.run.smithi156.stderr:103850K .......... .......... .......... .......... .......... 61% 27.7M 2s 2024-03-29T04:10:08.046 INFO:teuthology.orchestra.run.smithi156.stderr:103900K .......... .......... .......... .......... .......... 61% 22.7M 2s 2024-03-29T04:10:08.047 INFO:teuthology.orchestra.run.smithi156.stderr:103950K .......... .......... .......... .......... .......... 61% 26.7M 2s 2024-03-29T04:10:08.049 INFO:teuthology.orchestra.run.smithi156.stderr:104000K .......... .......... .......... .......... .......... 61% 27.1M 2s 2024-03-29T04:10:08.051 INFO:teuthology.orchestra.run.smithi156.stderr:104050K .......... .......... .......... .......... .......... 61% 27.3M 2s 2024-03-29T04:10:08.053 INFO:teuthology.orchestra.run.smithi156.stderr:104100K .......... .......... .......... .......... .......... 61% 23.3M 2s 2024-03-29T04:10:08.055 INFO:teuthology.orchestra.run.smithi156.stderr:104150K .......... .......... .......... .......... .......... 61% 26.8M 2s 2024-03-29T04:10:08.057 INFO:teuthology.orchestra.run.smithi156.stderr:104200K .......... .......... .......... .......... .......... 61% 26.6M 2s 2024-03-29T04:10:08.059 INFO:teuthology.orchestra.run.smithi156.stderr:104250K .......... .......... .......... .......... .......... 61% 26.9M 2s 2024-03-29T04:10:08.061 INFO:teuthology.orchestra.run.smithi156.stderr:104300K .......... .......... .......... .......... .......... 61% 23.0M 2s 2024-03-29T04:10:08.063 INFO:teuthology.orchestra.run.smithi156.stderr:104350K .......... .......... .......... .......... .......... 61% 26.7M 2s 2024-03-29T04:10:08.064 INFO:teuthology.orchestra.run.smithi156.stderr:104400K .......... .......... .......... .......... .......... 61% 26.4M 2s 2024-03-29T04:10:08.066 INFO:teuthology.orchestra.run.smithi156.stderr:104450K .......... .......... .......... .......... .......... 61% 26.5M 2s 2024-03-29T04:10:08.068 INFO:teuthology.orchestra.run.smithi156.stderr:104500K .......... .......... .......... .......... .......... 62% 22.6M 2s 2024-03-29T04:10:08.070 INFO:teuthology.orchestra.run.smithi156.stderr:104550K .......... .......... .......... .......... .......... 62% 25.9M 2s 2024-03-29T04:10:08.072 INFO:teuthology.orchestra.run.smithi156.stderr:104600K .......... .......... .......... .......... .......... 62% 26.1M 2s 2024-03-29T04:10:08.074 INFO:teuthology.orchestra.run.smithi156.stderr:104650K .......... .......... .......... .......... .......... 62% 26.1M 2s 2024-03-29T04:10:08.076 INFO:teuthology.orchestra.run.smithi156.stderr:104700K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-03-29T04:10:08.077 INFO:teuthology.orchestra.run.smithi156.stderr:104750K .......... .......... .......... .......... .......... 62% 33.2M 2s 2024-03-29T04:10:08.079 INFO:teuthology.orchestra.run.smithi156.stderr:104800K .......... .......... .......... .......... .......... 62% 33.6M 2s 2024-03-29T04:10:08.080 INFO:teuthology.orchestra.run.smithi156.stderr:104850K .......... .......... .......... .......... .......... 62% 34.1M 2s 2024-03-29T04:10:08.082 INFO:teuthology.orchestra.run.smithi156.stderr:104900K .......... .......... .......... .......... .......... 62% 28.3M 2s 2024-03-29T04:10:08.084 INFO:teuthology.orchestra.run.smithi156.stderr:104950K .......... .......... .......... .......... .......... 62% 28.5M 2s 2024-03-29T04:10:08.085 INFO:teuthology.orchestra.run.smithi156.stderr:105000K .......... .......... .......... .......... .......... 62% 28.3M 2s 2024-03-29T04:10:08.088 INFO:teuthology.orchestra.run.smithi156.stderr:105050K .......... .......... .......... .......... .......... 62% 28.3M 2s 2024-03-29T04:10:08.089 INFO:teuthology.orchestra.run.smithi156.stderr:105100K .......... .......... .......... .......... .......... 62% 24.6M 2s 2024-03-29T04:10:08.091 INFO:teuthology.orchestra.run.smithi156.stderr:105150K .......... .......... .......... .......... .......... 62% 27.1M 2s 2024-03-29T04:10:08.093 INFO:teuthology.orchestra.run.smithi156.stderr:105200K .......... .......... .......... .......... .......... 62% 25.2M 2s 2024-03-29T04:10:08.095 INFO:teuthology.orchestra.run.smithi156.stderr:105250K .......... .......... .......... .......... .......... 62% 25.0M 2s 2024-03-29T04:10:08.097 INFO:teuthology.orchestra.run.smithi156.stderr:105300K .......... .......... .......... .......... .......... 62% 21.5M 2s 2024-03-29T04:10:08.099 INFO:teuthology.orchestra.run.smithi156.stderr:105350K .......... .......... .......... .......... .......... 62% 25.3M 2s 2024-03-29T04:10:08.101 INFO:teuthology.orchestra.run.smithi156.stderr:105400K .......... .......... .......... .......... .......... 62% 25.6M 2s 2024-03-29T04:10:08.103 INFO:teuthology.orchestra.run.smithi156.stderr:105450K .......... .......... .......... .......... .......... 62% 25.7M 2s 2024-03-29T04:10:08.105 INFO:teuthology.orchestra.run.smithi156.stderr:105500K .......... .......... .......... .......... .......... 62% 21.9M 2s 2024-03-29T04:10:08.107 INFO:teuthology.orchestra.run.smithi156.stderr:105550K .......... .......... .......... .......... .......... 62% 25.7M 2s 2024-03-29T04:10:08.109 INFO:teuthology.orchestra.run.smithi156.stderr:105600K .......... .......... .......... .......... .......... 62% 25.1M 2s 2024-03-29T04:10:08.111 INFO:teuthology.orchestra.run.smithi156.stderr:105650K .......... .......... .......... .......... .......... 62% 24.7M 2s 2024-03-29T04:10:08.113 INFO:teuthology.orchestra.run.smithi156.stderr:105700K .......... .......... .......... .......... .......... 62% 21.7M 2s 2024-03-29T04:10:08.115 INFO:teuthology.orchestra.run.smithi156.stderr:105750K .......... .......... .......... .......... .......... 62% 24.8M 2s 2024-03-29T04:10:08.117 INFO:teuthology.orchestra.run.smithi156.stderr:105800K .......... .......... .......... .......... .......... 62% 24.5M 2s 2024-03-29T04:10:08.119 INFO:teuthology.orchestra.run.smithi156.stderr:105850K .......... .......... .......... .......... .......... 62% 24.8M 2s 2024-03-29T04:10:08.121 INFO:teuthology.orchestra.run.smithi156.stderr:105900K .......... .......... .......... .......... .......... 62% 21.8M 2s 2024-03-29T04:10:08.123 INFO:teuthology.orchestra.run.smithi156.stderr:105950K .......... .......... .......... .......... .......... 62% 24.7M 2s 2024-03-29T04:10:08.125 INFO:teuthology.orchestra.run.smithi156.stderr:106000K .......... .......... .......... .......... .......... 62% 24.3M 2s 2024-03-29T04:10:08.128 INFO:teuthology.orchestra.run.smithi156.stderr:106050K .......... .......... .......... .......... .......... 62% 23.8M 2s 2024-03-29T04:10:08.130 INFO:teuthology.orchestra.run.smithi156.stderr:106100K .......... .......... .......... .......... .......... 62% 22.0M 2s 2024-03-29T04:10:08.132 INFO:teuthology.orchestra.run.smithi156.stderr:106150K .......... .......... .......... .......... .......... 63% 25.3M 2s 2024-03-29T04:10:08.134 INFO:teuthology.orchestra.run.smithi156.stderr:106200K .......... .......... .......... .......... .......... 63% 25.6M 2s 2024-03-29T04:10:08.136 INFO:teuthology.orchestra.run.smithi156.stderr:106250K .......... .......... .......... .......... .......... 63% 24.5M 2s 2024-03-29T04:10:08.138 INFO:teuthology.orchestra.run.smithi156.stderr:106300K .......... .......... .......... .......... .......... 63% 22.0M 2s 2024-03-29T04:10:08.141 INFO:teuthology.orchestra.run.smithi156.stderr:106350K .......... .......... .......... .......... .......... 63% 24.2M 2s 2024-03-29T04:10:08.143 INFO:teuthology.orchestra.run.smithi156.stderr:106400K .......... .......... .......... .......... .......... 63% 25.1M 2s 2024-03-29T04:10:08.144 INFO:teuthology.orchestra.run.smithi156.stderr:106450K .......... .......... .......... .......... .......... 63% 25.2M 2s 2024-03-29T04:10:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:106500K .......... .......... .......... .......... .......... 63% 21.6M 2s 2024-03-29T04:10:08.148 INFO:teuthology.orchestra.run.smithi156.stderr:106550K .......... .......... .......... .......... .......... 63% 29.3M 2s 2024-03-29T04:10:08.149 INFO:teuthology.orchestra.run.smithi156.stderr:106600K .......... .......... .......... .......... .......... 63% 31.1M 2s 2024-03-29T04:10:08.151 INFO:teuthology.orchestra.run.smithi156.stderr:106650K .......... .......... .......... .......... .......... 63% 35.6M 2s 2024-03-29T04:10:08.152 INFO:teuthology.orchestra.run.smithi156.stderr:106700K .......... .......... .......... .......... .......... 63% 30.2M 2s 2024-03-29T04:10:08.154 INFO:teuthology.orchestra.run.smithi156.stderr:106750K .......... .......... .......... .......... .......... 63% 34.8M 2s 2024-03-29T04:10:08.159 INFO:teuthology.orchestra.run.smithi156.stderr:106800K .......... .......... .......... .......... .......... 63% 34.9M 2s 2024-03-29T04:10:08.159 INFO:teuthology.orchestra.run.smithi156.stderr:106850K .......... .......... .......... .......... .......... 63% 35.4M 2s 2024-03-29T04:10:08.159 INFO:teuthology.orchestra.run.smithi156.stderr:106900K .......... .......... .......... .......... .......... 63% 34.2M 2s 2024-03-29T04:10:08.161 INFO:teuthology.orchestra.run.smithi156.stderr:106950K .......... .......... .......... .......... .......... 63% 39.9M 2s 2024-03-29T04:10:08.161 INFO:teuthology.orchestra.run.smithi156.stderr:107000K .......... .......... .......... .......... .......... 63% 40.5M 2s 2024-03-29T04:10:08.162 INFO:teuthology.orchestra.run.smithi156.stderr:107050K .......... .......... .......... .......... .......... 63% 41.2M 2s 2024-03-29T04:10:08.163 INFO:teuthology.orchestra.run.smithi156.stderr:107100K .......... .......... .......... .......... .......... 63% 34.3M 2s 2024-03-29T04:10:08.164 INFO:teuthology.orchestra.run.smithi156.stderr:107150K .......... .......... .......... .......... .......... 63% 37.9M 2s 2024-03-29T04:10:08.165 INFO:teuthology.orchestra.run.smithi156.stderr:107200K .......... .......... .......... .......... .......... 63% 40.0M 2s 2024-03-29T04:10:08.167 INFO:teuthology.orchestra.run.smithi156.stderr:107250K .......... .......... .......... .......... .......... 63% 35.7M 2s 2024-03-29T04:10:08.169 INFO:teuthology.orchestra.run.smithi156.stderr:107300K .......... .......... .......... .......... .......... 63% 33.1M 2s 2024-03-29T04:10:08.169 INFO:teuthology.orchestra.run.smithi156.stderr:107350K .......... .......... .......... .......... .......... 63% 39.4M 2s 2024-03-29T04:10:08.171 INFO:teuthology.orchestra.run.smithi156.stderr:107400K .......... .......... .......... .......... .......... 63% 39.7M 2s 2024-03-29T04:10:08.172 INFO:teuthology.orchestra.run.smithi156.stderr:107450K .......... .......... .......... .......... .......... 63% 39.2M 2s 2024-03-29T04:10:08.173 INFO:teuthology.orchestra.run.smithi156.stderr:107500K .......... .......... .......... .......... .......... 63% 33.3M 2s 2024-03-29T04:10:08.175 INFO:teuthology.orchestra.run.smithi156.stderr:107550K .......... .......... .......... .......... .......... 63% 38.6M 2s 2024-03-29T04:10:08.176 INFO:teuthology.orchestra.run.smithi156.stderr:107600K .......... .......... .......... .......... .......... 63% 33.7M 2s 2024-03-29T04:10:08.178 INFO:teuthology.orchestra.run.smithi156.stderr:107650K .......... .......... .......... .......... .......... 63% 32.6M 2s 2024-03-29T04:10:08.179 INFO:teuthology.orchestra.run.smithi156.stderr:107700K .......... .......... .......... .......... .......... 63% 28.6M 2s 2024-03-29T04:10:08.181 INFO:teuthology.orchestra.run.smithi156.stderr:107750K .......... .......... .......... .......... .......... 63% 33.7M 2s 2024-03-29T04:10:08.182 INFO:teuthology.orchestra.run.smithi156.stderr:107800K .......... .......... .......... .......... .......... 63% 31.0M 2s 2024-03-29T04:10:08.184 INFO:teuthology.orchestra.run.smithi156.stderr:107850K .......... .......... .......... .......... .......... 64% 32.0M 2s 2024-03-29T04:10:08.186 INFO:teuthology.orchestra.run.smithi156.stderr:107900K .......... .......... .......... .......... .......... 64% 27.7M 2s 2024-03-29T04:10:08.187 INFO:teuthology.orchestra.run.smithi156.stderr:107950K .......... .......... .......... .......... .......... 64% 31.1M 2s 2024-03-29T04:10:08.189 INFO:teuthology.orchestra.run.smithi156.stderr:108000K .......... .......... .......... .......... .......... 64% 31.9M 2s 2024-03-29T04:10:08.190 INFO:teuthology.orchestra.run.smithi156.stderr:108050K .......... .......... .......... .......... .......... 64% 29.7M 2s 2024-03-29T04:10:08.192 INFO:teuthology.orchestra.run.smithi156.stderr:108100K .......... .......... .......... .......... .......... 64% 30.2M 2s 2024-03-29T04:10:08.193 INFO:teuthology.orchestra.run.smithi156.stderr:108150K .......... .......... .......... .......... .......... 64% 36.4M 2s 2024-03-29T04:10:08.195 INFO:teuthology.orchestra.run.smithi156.stderr:108200K .......... .......... .......... .......... .......... 64% 37.4M 2s 2024-03-29T04:10:08.196 INFO:teuthology.orchestra.run.smithi156.stderr:108250K .......... .......... .......... .......... .......... 64% 28.2M 2s 2024-03-29T04:10:08.198 INFO:teuthology.orchestra.run.smithi156.stderr:108300K .......... .......... .......... .......... .......... 64% 25.0M 2s 2024-03-29T04:10:08.200 INFO:teuthology.orchestra.run.smithi156.stderr:108350K .......... .......... .......... .......... .......... 64% 30.5M 2s 2024-03-29T04:10:08.201 INFO:teuthology.orchestra.run.smithi156.stderr:108400K .......... .......... .......... .......... .......... 64% 31.8M 2s 2024-03-29T04:10:08.203 INFO:teuthology.orchestra.run.smithi156.stderr:108450K .......... .......... .......... .......... .......... 64% 31.0M 2s 2024-03-29T04:10:08.205 INFO:teuthology.orchestra.run.smithi156.stderr:108500K .......... .......... .......... .......... .......... 64% 28.4M 2s 2024-03-29T04:10:08.206 INFO:teuthology.orchestra.run.smithi156.stderr:108550K .......... .......... .......... .......... .......... 64% 32.6M 2s 2024-03-29T04:10:08.207 INFO:teuthology.orchestra.run.smithi156.stderr:108600K .......... .......... .......... .......... .......... 64% 32.9M 2s 2024-03-29T04:10:08.209 INFO:teuthology.orchestra.run.smithi156.stderr:108650K .......... .......... .......... .......... .......... 64% 31.9M 2s 2024-03-29T04:10:08.211 INFO:teuthology.orchestra.run.smithi156.stderr:108700K .......... .......... .......... .......... .......... 64% 27.5M 2s 2024-03-29T04:10:08.212 INFO:teuthology.orchestra.run.smithi156.stderr:108750K .......... .......... .......... .......... .......... 64% 32.6M 2s 2024-03-29T04:10:08.214 INFO:teuthology.orchestra.run.smithi156.stderr:108800K .......... .......... .......... .......... .......... 64% 31.8M 2s 2024-03-29T04:10:08.216 INFO:teuthology.orchestra.run.smithi156.stderr:108850K .......... .......... .......... .......... .......... 64% 32.4M 2s 2024-03-29T04:10:08.217 INFO:teuthology.orchestra.run.smithi156.stderr:108900K .......... .......... .......... .......... .......... 64% 26.8M 2s 2024-03-29T04:10:08.219 INFO:teuthology.orchestra.run.smithi156.stderr:108950K .......... .......... .......... .......... .......... 64% 31.1M 2s 2024-03-29T04:10:08.220 INFO:teuthology.orchestra.run.smithi156.stderr:109000K .......... .......... .......... .......... .......... 64% 31.4M 2s 2024-03-29T04:10:08.222 INFO:teuthology.orchestra.run.smithi156.stderr:109050K .......... .......... .......... .......... .......... 64% 30.3M 2s 2024-03-29T04:10:08.224 INFO:teuthology.orchestra.run.smithi156.stderr:109100K .......... .......... .......... .......... .......... 64% 26.7M 2s 2024-03-29T04:10:08.226 INFO:teuthology.orchestra.run.smithi156.stderr:109150K .......... .......... .......... .......... .......... 64% 30.7M 2s 2024-03-29T04:10:08.227 INFO:teuthology.orchestra.run.smithi156.stderr:109200K .......... .......... .......... .......... .......... 64% 29.6M 2s 2024-03-29T04:10:08.229 INFO:teuthology.orchestra.run.smithi156.stderr:109250K .......... .......... .......... .......... .......... 64% 30.0M 2s 2024-03-29T04:10:08.231 INFO:teuthology.orchestra.run.smithi156.stderr:109300K .......... .......... .......... .......... .......... 64% 25.2M 2s 2024-03-29T04:10:08.233 INFO:teuthology.orchestra.run.smithi156.stderr:109350K .......... .......... .......... .......... .......... 64% 29.3M 2s 2024-03-29T04:10:08.234 INFO:teuthology.orchestra.run.smithi156.stderr:109400K .......... .......... .......... .......... .......... 64% 29.8M 2s 2024-03-29T04:10:08.237 INFO:teuthology.orchestra.run.smithi156.stderr:109450K .......... .......... .......... .......... .......... 64% 29.1M 2s 2024-03-29T04:10:08.240 INFO:teuthology.orchestra.run.smithi156.stderr:109500K .......... .......... .......... .......... .......... 64% 24.7M 2s 2024-03-29T04:10:08.240 INFO:teuthology.orchestra.run.smithi156.stderr:109550K .......... .......... .......... .......... .......... 65% 28.4M 2s 2024-03-29T04:10:08.242 INFO:teuthology.orchestra.run.smithi156.stderr:109600K .......... .......... .......... .......... .......... 65% 28.2M 2s 2024-03-29T04:10:08.243 INFO:teuthology.orchestra.run.smithi156.stderr:109650K .......... .......... .......... .......... .......... 65% 29.0M 2s 2024-03-29T04:10:08.245 INFO:teuthology.orchestra.run.smithi156.stderr:109700K .......... .......... .......... .......... .......... 65% 25.1M 2s 2024-03-29T04:10:08.246 INFO:teuthology.orchestra.run.smithi156.stderr:109750K .......... .......... .......... .......... .......... 65% 28.1M 2s 2024-03-29T04:10:08.249 INFO:teuthology.orchestra.run.smithi156.stderr:109800K .......... .......... .......... .......... .......... 65% 27.5M 2s 2024-03-29T04:10:08.251 INFO:teuthology.orchestra.run.smithi156.stderr:109850K .......... .......... .......... .......... .......... 65% 28.0M 2s 2024-03-29T04:10:08.253 INFO:teuthology.orchestra.run.smithi156.stderr:109900K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-03-29T04:10:08.255 INFO:teuthology.orchestra.run.smithi156.stderr:109950K .......... .......... .......... .......... .......... 65% 27.8M 2s 2024-03-29T04:10:08.256 INFO:teuthology.orchestra.run.smithi156.stderr:110000K .......... .......... .......... .......... .......... 65% 27.8M 2s 2024-03-29T04:10:08.258 INFO:teuthology.orchestra.run.smithi156.stderr:110050K .......... .......... .......... .......... .......... 65% 27.2M 2s 2024-03-29T04:10:08.259 INFO:teuthology.orchestra.run.smithi156.stderr:110100K .......... .......... .......... .......... .......... 65% 23.5M 2s 2024-03-29T04:10:08.261 INFO:teuthology.orchestra.run.smithi156.stderr:110150K .......... .......... .......... .......... .......... 65% 27.1M 2s 2024-03-29T04:10:08.263 INFO:teuthology.orchestra.run.smithi156.stderr:110200K .......... .......... .......... .......... .......... 65% 27.3M 2s 2024-03-29T04:10:08.265 INFO:teuthology.orchestra.run.smithi156.stderr:110250K .......... .......... .......... .......... .......... 65% 27.2M 2s 2024-03-29T04:10:08.267 INFO:teuthology.orchestra.run.smithi156.stderr:110300K .......... .......... .......... .......... .......... 65% 22.8M 2s 2024-03-29T04:10:08.269 INFO:teuthology.orchestra.run.smithi156.stderr:110350K .......... .......... .......... .......... .......... 65% 26.8M 2s 2024-03-29T04:10:08.273 INFO:teuthology.orchestra.run.smithi156.stderr:110400K .......... .......... .......... .......... .......... 65% 27.0M 2s 2024-03-29T04:10:08.273 INFO:teuthology.orchestra.run.smithi156.stderr:110450K .......... .......... .......... .......... .......... 65% 26.9M 2s 2024-03-29T04:10:08.275 INFO:teuthology.orchestra.run.smithi156.stderr:110500K .......... .......... .......... .......... .......... 65% 23.0M 2s 2024-03-29T04:10:08.276 INFO:teuthology.orchestra.run.smithi156.stderr:110550K .......... .......... .......... .......... .......... 65% 27.4M 2s 2024-03-29T04:10:08.278 INFO:teuthology.orchestra.run.smithi156.stderr:110600K .......... .......... .......... .......... .......... 65% 30.0M 2s 2024-03-29T04:10:08.280 INFO:teuthology.orchestra.run.smithi156.stderr:110650K .......... .......... .......... .......... .......... 65% 39.7M 2s 2024-03-29T04:10:08.285 INFO:teuthology.orchestra.run.smithi156.stderr:110700K .......... .......... .......... .......... .......... 65% 29.2M 2s 2024-03-29T04:10:08.285 INFO:teuthology.orchestra.run.smithi156.stderr:110750K .......... .......... .......... .......... .......... 65% 31.9M 2s 2024-03-29T04:10:08.285 INFO:teuthology.orchestra.run.smithi156.stderr:110800K .......... .......... .......... .......... .......... 65% 36.2M 2s 2024-03-29T04:10:08.285 INFO:teuthology.orchestra.run.smithi156.stderr:110850K .......... .......... .......... .......... .......... 65% 36.2M 2s 2024-03-29T04:10:08.287 INFO:teuthology.orchestra.run.smithi156.stderr:110900K .......... .......... .......... .......... .......... 65% 31.6M 2s 2024-03-29T04:10:08.288 INFO:teuthology.orchestra.run.smithi156.stderr:110950K .......... .......... .......... .......... .......... 65% 37.0M 2s 2024-03-29T04:10:08.289 INFO:teuthology.orchestra.run.smithi156.stderr:111000K .......... .......... .......... .......... .......... 65% 38.2M 2s 2024-03-29T04:10:08.290 INFO:teuthology.orchestra.run.smithi156.stderr:111050K .......... .......... .......... .......... .......... 65% 39.7M 2s 2024-03-29T04:10:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:111100K .......... .......... .......... .......... .......... 65% 34.0M 2s 2024-03-29T04:10:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:111150K .......... .......... .......... .......... .......... 65% 40.8M 2s 2024-03-29T04:10:08.295 INFO:teuthology.orchestra.run.smithi156.stderr:111200K .......... .......... .......... .......... .......... 65% 36.5M 2s 2024-03-29T04:10:08.296 INFO:teuthology.orchestra.run.smithi156.stderr:111250K .......... .......... .......... .......... .......... 66% 33.7M 2s 2024-03-29T04:10:08.297 INFO:teuthology.orchestra.run.smithi156.stderr:111300K .......... .......... .......... .......... .......... 66% 31.7M 2s 2024-03-29T04:10:08.299 INFO:teuthology.orchestra.run.smithi156.stderr:111350K .......... .......... .......... .......... .......... 66% 35.6M 2s 2024-03-29T04:10:08.300 INFO:teuthology.orchestra.run.smithi156.stderr:111400K .......... .......... .......... .......... .......... 66% 35.3M 2s 2024-03-29T04:10:08.301 INFO:teuthology.orchestra.run.smithi156.stderr:111450K .......... .......... .......... .......... .......... 66% 37.4M 2s 2024-03-29T04:10:08.303 INFO:teuthology.orchestra.run.smithi156.stderr:111500K .......... .......... .......... .......... .......... 66% 30.8M 2s 2024-03-29T04:10:08.304 INFO:teuthology.orchestra.run.smithi156.stderr:111550K .......... .......... .......... .......... .......... 66% 34.7M 2s 2024-03-29T04:10:08.307 INFO:teuthology.orchestra.run.smithi156.stderr:111600K .......... .......... .......... .......... .......... 66% 35.7M 2s 2024-03-29T04:10:08.307 INFO:teuthology.orchestra.run.smithi156.stderr:111650K .......... .......... .......... .......... .......... 66% 35.0M 2s 2024-03-29T04:10:08.309 INFO:teuthology.orchestra.run.smithi156.stderr:111700K .......... .......... .......... .......... .......... 66% 29.6M 2s 2024-03-29T04:10:08.310 INFO:teuthology.orchestra.run.smithi156.stderr:111750K .......... .......... .......... .......... .......... 66% 33.9M 2s 2024-03-29T04:10:08.312 INFO:teuthology.orchestra.run.smithi156.stderr:111800K .......... .......... .......... .......... .......... 66% 34.1M 2s 2024-03-29T04:10:08.313 INFO:teuthology.orchestra.run.smithi156.stderr:111850K .......... .......... .......... .......... .......... 66% 33.7M 2s 2024-03-29T04:10:08.315 INFO:teuthology.orchestra.run.smithi156.stderr:111900K .......... .......... .......... .......... .......... 66% 27.8M 2s 2024-03-29T04:10:08.316 INFO:teuthology.orchestra.run.smithi156.stderr:111950K .......... .......... .......... .......... .......... 66% 32.8M 2s 2024-03-29T04:10:08.318 INFO:teuthology.orchestra.run.smithi156.stderr:112000K .......... .......... .......... .......... .......... 66% 34.5M 2s 2024-03-29T04:10:08.320 INFO:teuthology.orchestra.run.smithi156.stderr:112050K .......... .......... .......... .......... .......... 66% 32.7M 2s 2024-03-29T04:10:08.322 INFO:teuthology.orchestra.run.smithi156.stderr:112100K .......... .......... .......... .......... .......... 66% 27.8M 2s 2024-03-29T04:10:08.323 INFO:teuthology.orchestra.run.smithi156.stderr:112150K .......... .......... .......... .......... .......... 66% 33.6M 2s 2024-03-29T04:10:08.324 INFO:teuthology.orchestra.run.smithi156.stderr:112200K .......... .......... .......... .......... .......... 66% 30.7M 2s 2024-03-29T04:10:08.326 INFO:teuthology.orchestra.run.smithi156.stderr:112250K .......... .......... .......... .......... .......... 66% 33.1M 2s 2024-03-29T04:10:08.328 INFO:teuthology.orchestra.run.smithi156.stderr:112300K .......... .......... .......... .......... .......... 66% 23.5M 2s 2024-03-29T04:10:08.329 INFO:teuthology.orchestra.run.smithi156.stderr:112350K .......... .......... .......... .......... .......... 66% 26.3M 2s 2024-03-29T04:10:08.331 INFO:teuthology.orchestra.run.smithi156.stderr:112400K .......... .......... .......... .......... .......... 66% 26.2M 2s 2024-03-29T04:10:08.333 INFO:teuthology.orchestra.run.smithi156.stderr:112450K .......... .......... .......... .......... .......... 66% 26.3M 2s 2024-03-29T04:10:08.335 INFO:teuthology.orchestra.run.smithi156.stderr:112500K .......... .......... .......... .......... .......... 66% 23.1M 2s 2024-03-29T04:10:08.337 INFO:teuthology.orchestra.run.smithi156.stderr:112550K .......... .......... .......... .......... .......... 66% 30.1M 2s 2024-03-29T04:10:08.338 INFO:teuthology.orchestra.run.smithi156.stderr:112600K .......... .......... .......... .......... .......... 66% 30.4M 2s 2024-03-29T04:10:08.340 INFO:teuthology.orchestra.run.smithi156.stderr:112650K .......... .......... .......... .......... .......... 66% 30.9M 2s 2024-03-29T04:10:08.342 INFO:teuthology.orchestra.run.smithi156.stderr:112700K .......... .......... .......... .......... .......... 66% 26.5M 2s 2024-03-29T04:10:08.344 INFO:teuthology.orchestra.run.smithi156.stderr:112750K .......... .......... .......... .......... .......... 66% 30.6M 2s 2024-03-29T04:10:08.345 INFO:teuthology.orchestra.run.smithi156.stderr:112800K .......... .......... .......... .......... .......... 66% 30.4M 2s 2024-03-29T04:10:08.347 INFO:teuthology.orchestra.run.smithi156.stderr:112850K .......... .......... .......... .......... .......... 66% 30.0M 2s 2024-03-29T04:10:08.349 INFO:teuthology.orchestra.run.smithi156.stderr:112900K .......... .......... .......... .......... .......... 67% 26.2M 2s 2024-03-29T04:10:08.350 INFO:teuthology.orchestra.run.smithi156.stderr:112950K .......... .......... .......... .......... .......... 67% 29.4M 2s 2024-03-29T04:10:08.352 INFO:teuthology.orchestra.run.smithi156.stderr:113000K .......... .......... .......... .......... .......... 67% 29.3M 2s 2024-03-29T04:10:08.354 INFO:teuthology.orchestra.run.smithi156.stderr:113050K .......... .......... .......... .......... .......... 67% 29.4M 2s 2024-03-29T04:10:08.356 INFO:teuthology.orchestra.run.smithi156.stderr:113100K .......... .......... .......... .......... .......... 67% 25.1M 2s 2024-03-29T04:10:08.357 INFO:teuthology.orchestra.run.smithi156.stderr:113150K .......... .......... .......... .......... .......... 67% 29.2M 2s 2024-03-29T04:10:08.359 INFO:teuthology.orchestra.run.smithi156.stderr:113200K .......... .......... .......... .......... .......... 67% 29.3M 2s 2024-03-29T04:10:08.361 INFO:teuthology.orchestra.run.smithi156.stderr:113250K .......... .......... .......... .......... .......... 67% 29.6M 2s 2024-03-29T04:10:08.363 INFO:teuthology.orchestra.run.smithi156.stderr:113300K .......... .......... .......... .......... .......... 67% 25.0M 2s 2024-03-29T04:10:08.366 INFO:teuthology.orchestra.run.smithi156.stderr:113350K .......... .......... .......... .......... .......... 67% 28.5M 2s 2024-03-29T04:10:08.366 INFO:teuthology.orchestra.run.smithi156.stderr:113400K .......... .......... .......... .......... .......... 67% 29.2M 2s 2024-03-29T04:10:08.368 INFO:teuthology.orchestra.run.smithi156.stderr:113450K .......... .......... .......... .......... .......... 67% 28.1M 2s 2024-03-29T04:10:08.370 INFO:teuthology.orchestra.run.smithi156.stderr:113500K .......... .......... .......... .......... .......... 67% 24.2M 2s 2024-03-29T04:10:08.371 INFO:teuthology.orchestra.run.smithi156.stderr:113550K .......... .......... .......... .......... .......... 67% 27.4M 2s 2024-03-29T04:10:08.373 INFO:teuthology.orchestra.run.smithi156.stderr:113600K .......... .......... .......... .......... .......... 67% 27.5M 2s 2024-03-29T04:10:08.375 INFO:teuthology.orchestra.run.smithi156.stderr:113650K .......... .......... .......... .......... .......... 67% 28.5M 2s 2024-03-29T04:10:08.377 INFO:teuthology.orchestra.run.smithi156.stderr:113700K .......... .......... .......... .......... .......... 67% 23.7M 2s 2024-03-29T04:10:08.379 INFO:teuthology.orchestra.run.smithi156.stderr:113750K .......... .......... .......... .......... .......... 67% 28.3M 2s 2024-03-29T04:10:08.380 INFO:teuthology.orchestra.run.smithi156.stderr:113800K .......... .......... .......... .......... .......... 67% 27.5M 2s 2024-03-29T04:10:08.382 INFO:teuthology.orchestra.run.smithi156.stderr:113850K .......... .......... .......... .......... .......... 67% 28.7M 2s 2024-03-29T04:10:08.384 INFO:teuthology.orchestra.run.smithi156.stderr:113900K .......... .......... .......... .......... .......... 67% 23.8M 2s 2024-03-29T04:10:08.386 INFO:teuthology.orchestra.run.smithi156.stderr:113950K .......... .......... .......... .......... .......... 67% 27.6M 2s 2024-03-29T04:10:08.388 INFO:teuthology.orchestra.run.smithi156.stderr:114000K .......... .......... .......... .......... .......... 67% 27.6M 2s 2024-03-29T04:10:08.390 INFO:teuthology.orchestra.run.smithi156.stderr:114050K .......... .......... .......... .......... .......... 67% 28.1M 2s 2024-03-29T04:10:08.392 INFO:teuthology.orchestra.run.smithi156.stderr:114100K .......... .......... .......... .......... .......... 67% 23.5M 2s 2024-03-29T04:10:08.393 INFO:teuthology.orchestra.run.smithi156.stderr:114150K .......... .......... .......... .......... .......... 67% 26.8M 2s 2024-03-29T04:10:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:114200K .......... .......... .......... .......... .......... 67% 25.8M 2s 2024-03-29T04:10:08.398 INFO:teuthology.orchestra.run.smithi156.stderr:114250K .......... .......... .......... .......... .......... 67% 25.2M 2s 2024-03-29T04:10:08.400 INFO:teuthology.orchestra.run.smithi156.stderr:114300K .......... .......... .......... .......... .......... 67% 21.1M 2s 2024-03-29T04:10:08.402 INFO:teuthology.orchestra.run.smithi156.stderr:114350K .......... .......... .......... .......... .......... 67% 24.3M 2s 2024-03-29T04:10:08.403 INFO:teuthology.orchestra.run.smithi156.stderr:114400K .......... .......... .......... .......... .......... 67% 28.5M 2s 2024-03-29T04:10:08.405 INFO:teuthology.orchestra.run.smithi156.stderr:114450K .......... .......... .......... .......... .......... 67% 29.0M 2s 2024-03-29T04:10:08.407 INFO:teuthology.orchestra.run.smithi156.stderr:114500K .......... .......... .......... .......... .......... 67% 25.0M 2s 2024-03-29T04:10:08.409 INFO:teuthology.orchestra.run.smithi156.stderr:114550K .......... .......... .......... .......... .......... 67% 28.6M 2s 2024-03-29T04:10:08.411 INFO:teuthology.orchestra.run.smithi156.stderr:114600K .......... .......... .......... .......... .......... 68% 28.3M 2s 2024-03-29T04:10:08.413 INFO:teuthology.orchestra.run.smithi156.stderr:114650K .......... .......... .......... .......... .......... 68% 28.8M 2s 2024-03-29T04:10:08.414 INFO:teuthology.orchestra.run.smithi156.stderr:114700K .......... .......... .......... .......... .......... 68% 26.0M 2s 2024-03-29T04:10:08.417 INFO:teuthology.orchestra.run.smithi156.stderr:114750K .......... .......... .......... .......... .......... 68% 34.3M 2s 2024-03-29T04:10:08.418 INFO:teuthology.orchestra.run.smithi156.stderr:114800K .......... .......... .......... .......... .......... 68% 25.4M 2s 2024-03-29T04:10:08.420 INFO:teuthology.orchestra.run.smithi156.stderr:114850K .......... .......... .......... .......... .......... 68% 30.4M 2s 2024-03-29T04:10:08.422 INFO:teuthology.orchestra.run.smithi156.stderr:114900K .......... .......... .......... .......... .......... 68% 26.1M 2s 2024-03-29T04:10:08.423 INFO:teuthology.orchestra.run.smithi156.stderr:114950K .......... .......... .......... .......... .......... 68% 32.4M 2s 2024-03-29T04:10:08.424 INFO:teuthology.orchestra.run.smithi156.stderr:115000K .......... .......... .......... .......... .......... 68% 35.6M 2s 2024-03-29T04:10:08.425 INFO:teuthology.orchestra.run.smithi156.stderr:115050K .......... .......... .......... .......... .......... 68% 37.9M 2s 2024-03-29T04:10:08.426 INFO:teuthology.orchestra.run.smithi156.stderr:115100K .......... .......... .......... .......... .......... 68% 32.6M 2s 2024-03-29T04:10:08.428 INFO:teuthology.orchestra.run.smithi156.stderr:115150K .......... .......... .......... .......... .......... 68% 31.4M 2s 2024-03-29T04:10:08.430 INFO:teuthology.orchestra.run.smithi156.stderr:115200K .......... .......... .......... .......... .......... 68% 32.3M 2s 2024-03-29T04:10:08.431 INFO:teuthology.orchestra.run.smithi156.stderr:115250K .......... .......... .......... .......... .......... 68% 37.7M 2s 2024-03-29T04:10:08.433 INFO:teuthology.orchestra.run.smithi156.stderr:115300K .......... .......... .......... .......... .......... 68% 31.4M 2s 2024-03-29T04:10:08.434 INFO:teuthology.orchestra.run.smithi156.stderr:115350K .......... .......... .......... .......... .......... 68% 39.1M 2s 2024-03-29T04:10:08.435 INFO:teuthology.orchestra.run.smithi156.stderr:115400K .......... .......... .......... .......... .......... 68% 36.8M 2s 2024-03-29T04:10:08.438 INFO:teuthology.orchestra.run.smithi156.stderr:115450K .......... .......... .......... .......... .......... 68% 34.6M 2s 2024-03-29T04:10:08.438 INFO:teuthology.orchestra.run.smithi156.stderr:115500K .......... .......... .......... .......... .......... 68% 31.9M 2s 2024-03-29T04:10:08.440 INFO:teuthology.orchestra.run.smithi156.stderr:115550K .......... .......... .......... .......... .......... 68% 34.1M 2s 2024-03-29T04:10:08.441 INFO:teuthology.orchestra.run.smithi156.stderr:115600K .......... .......... .......... .......... .......... 68% 36.5M 2s 2024-03-29T04:10:08.443 INFO:teuthology.orchestra.run.smithi156.stderr:115650K .......... .......... .......... .......... .......... 68% 36.3M 2s 2024-03-29T04:10:08.444 INFO:teuthology.orchestra.run.smithi156.stderr:115700K .......... .......... .......... .......... .......... 68% 29.7M 2s 2024-03-29T04:10:08.446 INFO:teuthology.orchestra.run.smithi156.stderr:115750K .......... .......... .......... .......... .......... 68% 35.7M 2s 2024-03-29T04:10:08.447 INFO:teuthology.orchestra.run.smithi156.stderr:115800K .......... .......... .......... .......... .......... 68% 36.0M 2s 2024-03-29T04:10:08.448 INFO:teuthology.orchestra.run.smithi156.stderr:115850K .......... .......... .......... .......... .......... 68% 36.7M 2s 2024-03-29T04:10:08.449 INFO:teuthology.orchestra.run.smithi156.stderr:115900K .......... .......... .......... .......... .......... 68% 31.7M 2s 2024-03-29T04:10:08.451 INFO:teuthology.orchestra.run.smithi156.stderr:115950K .......... .......... .......... .......... .......... 68% 37.8M 2s 2024-03-29T04:10:08.452 INFO:teuthology.orchestra.run.smithi156.stderr:116000K .......... .......... .......... .......... .......... 68% 37.6M 2s 2024-03-29T04:10:08.453 INFO:teuthology.orchestra.run.smithi156.stderr:116050K .......... .......... .......... .......... .......... 68% 37.1M 2s 2024-03-29T04:10:08.455 INFO:teuthology.orchestra.run.smithi156.stderr:116100K .......... .......... .......... .......... .......... 68% 31.8M 2s 2024-03-29T04:10:08.456 INFO:teuthology.orchestra.run.smithi156.stderr:116150K .......... .......... .......... .......... .......... 68% 35.9M 2s 2024-03-29T04:10:08.457 INFO:teuthology.orchestra.run.smithi156.stderr:116200K .......... .......... .......... .......... .......... 68% 35.5M 2s 2024-03-29T04:10:08.459 INFO:teuthology.orchestra.run.smithi156.stderr:116250K .......... .......... .......... .......... .......... 68% 35.4M 2s 2024-03-29T04:10:08.460 INFO:teuthology.orchestra.run.smithi156.stderr:116300K .......... .......... .......... .......... .......... 69% 31.2M 2s 2024-03-29T04:10:08.462 INFO:teuthology.orchestra.run.smithi156.stderr:116350K .......... .......... .......... .......... .......... 69% 35.6M 2s 2024-03-29T04:10:08.463 INFO:teuthology.orchestra.run.smithi156.stderr:116400K .......... .......... .......... .......... .......... 69% 35.6M 2s 2024-03-29T04:10:08.465 INFO:teuthology.orchestra.run.smithi156.stderr:116450K .......... .......... .......... .......... .......... 69% 34.0M 2s 2024-03-29T04:10:08.466 INFO:teuthology.orchestra.run.smithi156.stderr:116500K .......... .......... .......... .......... .......... 69% 29.5M 2s 2024-03-29T04:10:08.467 INFO:teuthology.orchestra.run.smithi156.stderr:116550K .......... .......... .......... .......... .......... 69% 34.0M 2s 2024-03-29T04:10:08.469 INFO:teuthology.orchestra.run.smithi156.stderr:116600K .......... .......... .......... .......... .......... 69% 32.3M 2s 2024-03-29T04:10:08.471 INFO:teuthology.orchestra.run.smithi156.stderr:116650K .......... .......... .......... .......... .......... 69% 33.6M 2s 2024-03-29T04:10:08.472 INFO:teuthology.orchestra.run.smithi156.stderr:116700K .......... .......... .......... .......... .......... 69% 29.0M 2s 2024-03-29T04:10:08.474 INFO:teuthology.orchestra.run.smithi156.stderr:116750K .......... .......... .......... .......... .......... 69% 35.0M 2s 2024-03-29T04:10:08.475 INFO:teuthology.orchestra.run.smithi156.stderr:116800K .......... .......... .......... .......... .......... 69% 34.2M 2s 2024-03-29T04:10:08.477 INFO:teuthology.orchestra.run.smithi156.stderr:116850K .......... .......... .......... .......... .......... 69% 32.9M 2s 2024-03-29T04:10:08.478 INFO:teuthology.orchestra.run.smithi156.stderr:116900K .......... .......... .......... .......... .......... 69% 28.8M 2s 2024-03-29T04:10:08.479 INFO:teuthology.orchestra.run.smithi156.stderr:116950K .......... .......... .......... .......... .......... 69% 35.0M 2s 2024-03-29T04:10:08.481 INFO:teuthology.orchestra.run.smithi156.stderr:117000K .......... .......... .......... .......... .......... 69% 33.2M 2s 2024-03-29T04:10:08.483 INFO:teuthology.orchestra.run.smithi156.stderr:117050K .......... .......... .......... .......... .......... 69% 33.2M 2s 2024-03-29T04:10:08.484 INFO:teuthology.orchestra.run.smithi156.stderr:117100K .......... .......... .......... .......... .......... 69% 29.6M 2s 2024-03-29T04:10:08.486 INFO:teuthology.orchestra.run.smithi156.stderr:117150K .......... .......... .......... .......... .......... 69% 32.2M 2s 2024-03-29T04:10:08.487 INFO:teuthology.orchestra.run.smithi156.stderr:117200K .......... .......... .......... .......... .......... 69% 33.8M 2s 2024-03-29T04:10:08.489 INFO:teuthology.orchestra.run.smithi156.stderr:117250K .......... .......... .......... .......... .......... 69% 34.2M 2s 2024-03-29T04:10:08.490 INFO:teuthology.orchestra.run.smithi156.stderr:117300K .......... .......... .......... .......... .......... 69% 27.7M 2s 2024-03-29T04:10:08.492 INFO:teuthology.orchestra.run.smithi156.stderr:117350K .......... .......... .......... .......... .......... 69% 32.9M 2s 2024-03-29T04:10:08.493 INFO:teuthology.orchestra.run.smithi156.stderr:117400K .......... .......... .......... .......... .......... 69% 32.4M 2s 2024-03-29T04:10:08.495 INFO:teuthology.orchestra.run.smithi156.stderr:117450K .......... .......... .......... .......... .......... 69% 32.9M 2s 2024-03-29T04:10:08.497 INFO:teuthology.orchestra.run.smithi156.stderr:117500K .......... .......... .......... .......... .......... 69% 26.1M 2s 2024-03-29T04:10:08.498 INFO:teuthology.orchestra.run.smithi156.stderr:117550K .......... .......... .......... .......... .......... 69% 30.1M 2s 2024-03-29T04:10:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:117600K .......... .......... .......... .......... .......... 69% 30.9M 2s 2024-03-29T04:10:08.502 INFO:teuthology.orchestra.run.smithi156.stderr:117650K .......... .......... .......... .......... .......... 69% 30.9M 2s 2024-03-29T04:10:08.503 INFO:teuthology.orchestra.run.smithi156.stderr:117700K .......... .......... .......... .......... .......... 69% 26.6M 2s 2024-03-29T04:10:08.505 INFO:teuthology.orchestra.run.smithi156.stderr:117750K .......... .......... .......... .......... .......... 69% 30.6M 2s 2024-03-29T04:10:08.508 INFO:teuthology.orchestra.run.smithi156.stderr:117800K .......... .......... .......... .......... .......... 69% 31.7M 2s 2024-03-29T04:10:08.508 INFO:teuthology.orchestra.run.smithi156.stderr:117850K .......... .......... .......... .......... .......... 69% 31.3M 2s 2024-03-29T04:10:08.510 INFO:teuthology.orchestra.run.smithi156.stderr:117900K .......... .......... .......... .......... .......... 69% 26.8M 2s 2024-03-29T04:10:08.511 INFO:teuthology.orchestra.run.smithi156.stderr:117950K .......... .......... .......... .......... .......... 70% 31.4M 2s 2024-03-29T04:10:08.513 INFO:teuthology.orchestra.run.smithi156.stderr:118000K .......... .......... .......... .......... .......... 70% 29.7M 2s 2024-03-29T04:10:08.515 INFO:teuthology.orchestra.run.smithi156.stderr:118050K .......... .......... .......... .......... .......... 70% 29.5M 2s 2024-03-29T04:10:08.517 INFO:teuthology.orchestra.run.smithi156.stderr:118100K .......... .......... .......... .......... .......... 70% 25.8M 2s 2024-03-29T04:10:08.518 INFO:teuthology.orchestra.run.smithi156.stderr:118150K .......... .......... .......... .......... .......... 70% 29.7M 2s 2024-03-29T04:10:08.520 INFO:teuthology.orchestra.run.smithi156.stderr:118200K .......... .......... .......... .......... .......... 70% 30.8M 2s 2024-03-29T04:10:08.522 INFO:teuthology.orchestra.run.smithi156.stderr:118250K .......... .......... .......... .......... .......... 70% 30.2M 2s 2024-03-29T04:10:08.524 INFO:teuthology.orchestra.run.smithi156.stderr:118300K .......... .......... .......... .......... .......... 70% 26.2M 2s 2024-03-29T04:10:08.525 INFO:teuthology.orchestra.run.smithi156.stderr:118350K .......... .......... .......... .......... .......... 70% 29.9M 2s 2024-03-29T04:10:08.527 INFO:teuthology.orchestra.run.smithi156.stderr:118400K .......... .......... .......... .......... .......... 70% 29.0M 2s 2024-03-29T04:10:08.528 INFO:teuthology.orchestra.run.smithi156.stderr:118450K .......... .......... .......... .......... .......... 70% 29.9M 2s 2024-03-29T04:10:08.530 INFO:teuthology.orchestra.run.smithi156.stderr:118500K .......... .......... .......... .......... .......... 70% 25.6M 2s 2024-03-29T04:10:08.532 INFO:teuthology.orchestra.run.smithi156.stderr:118550K .......... .......... .......... .......... .......... 70% 30.4M 2s 2024-03-29T04:10:08.533 INFO:teuthology.orchestra.run.smithi156.stderr:118600K .......... .......... .......... .......... .......... 70% 29.5M 2s 2024-03-29T04:10:08.536 INFO:teuthology.orchestra.run.smithi156.stderr:118650K .......... .......... .......... .......... .......... 70% 27.6M 2s 2024-03-29T04:10:08.537 INFO:teuthology.orchestra.run.smithi156.stderr:118700K .......... .......... .......... .......... .......... 70% 25.3M 2s 2024-03-29T04:10:08.539 INFO:teuthology.orchestra.run.smithi156.stderr:118750K .......... .......... .......... .......... .......... 70% 27.1M 2s 2024-03-29T04:10:08.540 INFO:teuthology.orchestra.run.smithi156.stderr:118800K .......... .......... .......... .......... .......... 70% 36.0M 2s 2024-03-29T04:10:08.542 INFO:teuthology.orchestra.run.smithi156.stderr:118850K .......... .......... .......... .......... .......... 70% 35.1M 2s 2024-03-29T04:10:08.544 INFO:teuthology.orchestra.run.smithi156.stderr:118900K .......... .......... .......... .......... .......... 70% 24.3M 2s 2024-03-29T04:10:08.546 INFO:teuthology.orchestra.run.smithi156.stderr:118950K .......... .......... .......... .......... .......... 70% 26.4M 2s 2024-03-29T04:10:08.547 INFO:teuthology.orchestra.run.smithi156.stderr:119000K .......... .......... .......... .......... .......... 70% 28.2M 2s 2024-03-29T04:10:08.549 INFO:teuthology.orchestra.run.smithi156.stderr:119050K .......... .......... .......... .......... .......... 70% 33.9M 2s 2024-03-29T04:10:08.550 INFO:teuthology.orchestra.run.smithi156.stderr:119100K .......... .......... .......... .......... .......... 70% 30.0M 2s 2024-03-29T04:10:08.552 INFO:teuthology.orchestra.run.smithi156.stderr:119150K .......... .......... .......... .......... .......... 70% 30.4M 2s 2024-03-29T04:10:08.553 INFO:teuthology.orchestra.run.smithi156.stderr:119200K .......... .......... .......... .......... .......... 70% 38.2M 2s 2024-03-29T04:10:08.555 INFO:teuthology.orchestra.run.smithi156.stderr:119250K .......... .......... .......... .......... .......... 70% 37.8M 2s 2024-03-29T04:10:08.556 INFO:teuthology.orchestra.run.smithi156.stderr:119300K .......... .......... .......... .......... .......... 70% 30.9M 2s 2024-03-29T04:10:08.557 INFO:teuthology.orchestra.run.smithi156.stderr:119350K .......... .......... .......... .......... .......... 70% 35.7M 2s 2024-03-29T04:10:08.559 INFO:teuthology.orchestra.run.smithi156.stderr:119400K .......... .......... .......... .......... .......... 70% 33.2M 2s 2024-03-29T04:10:08.561 INFO:teuthology.orchestra.run.smithi156.stderr:119450K .......... .......... .......... .......... .......... 70% 33.3M 2s 2024-03-29T04:10:08.562 INFO:teuthology.orchestra.run.smithi156.stderr:119500K .......... .......... .......... .......... .......... 70% 27.9M 2s 2024-03-29T04:10:08.564 INFO:teuthology.orchestra.run.smithi156.stderr:119550K .......... .......... .......... .......... .......... 70% 34.5M 2s 2024-03-29T04:10:08.565 INFO:teuthology.orchestra.run.smithi156.stderr:119600K .......... .......... .......... .......... .......... 70% 33.3M 2s 2024-03-29T04:10:08.567 INFO:teuthology.orchestra.run.smithi156.stderr:119650K .......... .......... .......... .......... .......... 71% 33.3M 2s 2024-03-29T04:10:08.571 INFO:teuthology.orchestra.run.smithi156.stderr:119700K .......... .......... .......... .......... .......... 71% 28.5M 2s 2024-03-29T04:10:08.571 INFO:teuthology.orchestra.run.smithi156.stderr:119750K .......... .......... .......... .......... .......... 71% 33.7M 2s 2024-03-29T04:10:08.572 INFO:teuthology.orchestra.run.smithi156.stderr:119800K .......... .......... .......... .......... .......... 71% 32.9M 2s 2024-03-29T04:10:08.573 INFO:teuthology.orchestra.run.smithi156.stderr:119850K .......... .......... .......... .......... .......... 71% 31.7M 2s 2024-03-29T04:10:08.575 INFO:teuthology.orchestra.run.smithi156.stderr:119900K .......... .......... .......... .......... .......... 71% 27.2M 2s 2024-03-29T04:10:08.576 INFO:teuthology.orchestra.run.smithi156.stderr:119950K .......... .......... .......... .......... .......... 71% 32.3M 2s 2024-03-29T04:10:08.578 INFO:teuthology.orchestra.run.smithi156.stderr:120000K .......... .......... .......... .......... .......... 71% 32.5M 2s 2024-03-29T04:10:08.579 INFO:teuthology.orchestra.run.smithi156.stderr:120050K .......... .......... .......... .......... .......... 71% 31.6M 2s 2024-03-29T04:10:08.581 INFO:teuthology.orchestra.run.smithi156.stderr:120100K .......... .......... .......... .......... .......... 71% 27.9M 2s 2024-03-29T04:10:08.582 INFO:teuthology.orchestra.run.smithi156.stderr:120150K .......... .......... .......... .......... .......... 71% 32.5M 2s 2024-03-29T04:10:08.584 INFO:teuthology.orchestra.run.smithi156.stderr:120200K .......... .......... .......... .......... .......... 71% 32.0M 2s 2024-03-29T04:10:08.585 INFO:teuthology.orchestra.run.smithi156.stderr:120250K .......... .......... .......... .......... .......... 71% 31.0M 2s 2024-03-29T04:10:08.587 INFO:teuthology.orchestra.run.smithi156.stderr:120300K .......... .......... .......... .......... .......... 71% 26.8M 2s 2024-03-29T04:10:08.589 INFO:teuthology.orchestra.run.smithi156.stderr:120350K .......... .......... .......... .......... .......... 71% 31.1M 2s 2024-03-29T04:10:08.590 INFO:teuthology.orchestra.run.smithi156.stderr:120400K .......... .......... .......... .......... .......... 71% 30.3M 2s 2024-03-29T04:10:08.592 INFO:teuthology.orchestra.run.smithi156.stderr:120450K .......... .......... .......... .......... .......... 71% 30.9M 2s 2024-03-29T04:10:08.594 INFO:teuthology.orchestra.run.smithi156.stderr:120500K .......... .......... .......... .......... .......... 71% 26.7M 2s 2024-03-29T04:10:08.596 INFO:teuthology.orchestra.run.smithi156.stderr:120550K .......... .......... .......... .......... .......... 71% 29.7M 2s 2024-03-29T04:10:08.597 INFO:teuthology.orchestra.run.smithi156.stderr:120600K .......... .......... .......... .......... .......... 71% 31.6M 2s 2024-03-29T04:10:08.599 INFO:teuthology.orchestra.run.smithi156.stderr:120650K .......... .......... .......... .......... .......... 71% 33.5M 2s 2024-03-29T04:10:08.600 INFO:teuthology.orchestra.run.smithi156.stderr:120700K .......... .......... .......... .......... .......... 71% 28.0M 2s 2024-03-29T04:10:08.602 INFO:teuthology.orchestra.run.smithi156.stderr:120750K .......... .......... .......... .......... .......... 71% 33.6M 2s 2024-03-29T04:10:08.603 INFO:teuthology.orchestra.run.smithi156.stderr:120800K .......... .......... .......... .......... .......... 71% 33.7M 2s 2024-03-29T04:10:08.605 INFO:teuthology.orchestra.run.smithi156.stderr:120850K .......... .......... .......... .......... .......... 71% 32.9M 2s 2024-03-29T04:10:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:120900K .......... .......... .......... .......... .......... 71% 27.9M 2s 2024-03-29T04:10:08.608 INFO:teuthology.orchestra.run.smithi156.stderr:120950K .......... .......... .......... .......... .......... 71% 33.1M 2s 2024-03-29T04:10:08.609 INFO:teuthology.orchestra.run.smithi156.stderr:121000K .......... .......... .......... .......... .......... 71% 33.4M 2s 2024-03-29T04:10:08.611 INFO:teuthology.orchestra.run.smithi156.stderr:121050K .......... .......... .......... .......... .......... 71% 32.8M 2s 2024-03-29T04:10:08.614 INFO:teuthology.orchestra.run.smithi156.stderr:121100K .......... .......... .......... .......... .......... 71% 27.6M 2s 2024-03-29T04:10:08.614 INFO:teuthology.orchestra.run.smithi156.stderr:121150K .......... .......... .......... .......... .......... 71% 32.3M 2s 2024-03-29T04:10:08.616 INFO:teuthology.orchestra.run.smithi156.stderr:121200K .......... .......... .......... .......... .......... 71% 32.9M 2s 2024-03-29T04:10:08.617 INFO:teuthology.orchestra.run.smithi156.stderr:121250K .......... .......... .......... .......... .......... 71% 31.7M 2s 2024-03-29T04:10:08.619 INFO:teuthology.orchestra.run.smithi156.stderr:121300K .......... .......... .......... .......... .......... 71% 27.7M 2s 2024-03-29T04:10:08.620 INFO:teuthology.orchestra.run.smithi156.stderr:121350K .......... .......... .......... .......... .......... 72% 32.4M 2s 2024-03-29T04:10:08.622 INFO:teuthology.orchestra.run.smithi156.stderr:121400K .......... .......... .......... .......... .......... 72% 32.2M 2s 2024-03-29T04:10:08.624 INFO:teuthology.orchestra.run.smithi156.stderr:121450K .......... .......... .......... .......... .......... 72% 31.5M 2s 2024-03-29T04:10:08.626 INFO:teuthology.orchestra.run.smithi156.stderr:121500K .......... .......... .......... .......... .......... 72% 27.5M 2s 2024-03-29T04:10:08.627 INFO:teuthology.orchestra.run.smithi156.stderr:121550K .......... .......... .......... .......... .......... 72% 29.1M 2s 2024-03-29T04:10:08.628 INFO:teuthology.orchestra.run.smithi156.stderr:121600K .......... .......... .......... .......... .......... 72% 31.0M 2s 2024-03-29T04:10:08.630 INFO:teuthology.orchestra.run.smithi156.stderr:121650K .......... .......... .......... .......... .......... 72% 29.5M 2s 2024-03-29T04:10:08.632 INFO:teuthology.orchestra.run.smithi156.stderr:121700K .......... .......... .......... .......... .......... 72% 25.9M 2s 2024-03-29T04:10:08.634 INFO:teuthology.orchestra.run.smithi156.stderr:121750K .......... .......... .......... .......... .......... 72% 30.5M 2s 2024-03-29T04:10:08.635 INFO:teuthology.orchestra.run.smithi156.stderr:121800K .......... .......... .......... .......... .......... 72% 29.5M 2s 2024-03-29T04:10:08.637 INFO:teuthology.orchestra.run.smithi156.stderr:121850K .......... .......... .......... .......... .......... 72% 30.1M 2s 2024-03-29T04:10:08.639 INFO:teuthology.orchestra.run.smithi156.stderr:121900K .......... .......... .......... .......... .......... 72% 24.9M 2s 2024-03-29T04:10:08.641 INFO:teuthology.orchestra.run.smithi156.stderr:121950K .......... .......... .......... .......... .......... 72% 30.2M 2s 2024-03-29T04:10:08.642 INFO:teuthology.orchestra.run.smithi156.stderr:122000K .......... .......... .......... .......... .......... 72% 27.2M 2s 2024-03-29T04:10:08.644 INFO:teuthology.orchestra.run.smithi156.stderr:122050K .......... .......... .......... .......... .......... 72% 25.9M 2s 2024-03-29T04:10:08.646 INFO:teuthology.orchestra.run.smithi156.stderr:122100K .......... .......... .......... .......... .......... 72% 25.2M 2s 2024-03-29T04:10:08.648 INFO:teuthology.orchestra.run.smithi156.stderr:122150K .......... .......... .......... .......... .......... 72% 30.3M 2s 2024-03-29T04:10:08.649 INFO:teuthology.orchestra.run.smithi156.stderr:122200K .......... .......... .......... .......... .......... 72% 29.9M 2s 2024-03-29T04:10:08.651 INFO:teuthology.orchestra.run.smithi156.stderr:122250K .......... .......... .......... .......... .......... 72% 30.1M 2s 2024-03-29T04:10:08.653 INFO:teuthology.orchestra.run.smithi156.stderr:122300K .......... .......... .......... .......... .......... 72% 25.9M 2s 2024-03-29T04:10:08.654 INFO:teuthology.orchestra.run.smithi156.stderr:122350K .......... .......... .......... .......... .......... 72% 30.1M 2s 2024-03-29T04:10:08.656 INFO:teuthology.orchestra.run.smithi156.stderr:122400K .......... .......... .......... .......... .......... 72% 24.2M 2s 2024-03-29T04:10:08.658 INFO:teuthology.orchestra.run.smithi156.stderr:122450K .......... .......... .......... .......... .......... 72% 26.6M 2s 2024-03-29T04:10:08.661 INFO:teuthology.orchestra.run.smithi156.stderr:122500K .......... .......... .......... .......... .......... 72% 24.3M 2s 2024-03-29T04:10:08.662 INFO:teuthology.orchestra.run.smithi156.stderr:122550K .......... .......... .......... .......... .......... 72% 28.1M 2s 2024-03-29T04:10:08.664 INFO:teuthology.orchestra.run.smithi156.stderr:122600K .......... .......... .......... .......... .......... 72% 26.6M 2s 2024-03-29T04:10:08.666 INFO:teuthology.orchestra.run.smithi156.stderr:122650K .......... .......... .......... .......... .......... 72% 27.4M 2s 2024-03-29T04:10:08.668 INFO:teuthology.orchestra.run.smithi156.stderr:122700K .......... .......... .......... .......... .......... 72% 23.7M 2s 2024-03-29T04:10:08.670 INFO:teuthology.orchestra.run.smithi156.stderr:122750K .......... .......... .......... .......... .......... 72% 26.5M 2s 2024-03-29T04:10:08.671 INFO:teuthology.orchestra.run.smithi156.stderr:122800K .......... .......... .......... .......... .......... 72% 27.1M 2s 2024-03-29T04:10:08.674 INFO:teuthology.orchestra.run.smithi156.stderr:122850K .......... .......... .......... .......... .......... 72% 23.8M 2s 2024-03-29T04:10:08.676 INFO:teuthology.orchestra.run.smithi156.stderr:122900K .......... .......... .......... .......... .......... 72% 23.3M 2s 2024-03-29T04:10:08.678 INFO:teuthology.orchestra.run.smithi156.stderr:122950K .......... .......... .......... .......... .......... 72% 23.4M 2s 2024-03-29T04:10:08.680 INFO:teuthology.orchestra.run.smithi156.stderr:123000K .......... .......... .......... .......... .......... 72% 23.8M 2s 2024-03-29T04:10:08.682 INFO:teuthology.orchestra.run.smithi156.stderr:123050K .......... .......... .......... .......... .......... 73% 28.3M 2s 2024-03-29T04:10:08.683 INFO:teuthology.orchestra.run.smithi156.stderr:123100K .......... .......... .......... .......... .......... 73% 28.4M 2s 2024-03-29T04:10:08.685 INFO:teuthology.orchestra.run.smithi156.stderr:123150K .......... .......... .......... .......... .......... 73% 28.3M 2s 2024-03-29T04:10:08.686 INFO:teuthology.orchestra.run.smithi156.stderr:123200K .......... .......... .......... .......... .......... 73% 41.2M 2s 2024-03-29T04:10:08.687 INFO:teuthology.orchestra.run.smithi156.stderr:123250K .......... .......... .......... .......... .......... 73% 43.0M 2s 2024-03-29T04:10:08.688 INFO:teuthology.orchestra.run.smithi156.stderr:123300K .......... .......... .......... .......... .......... 73% 37.3M 2s 2024-03-29T04:10:08.690 INFO:teuthology.orchestra.run.smithi156.stderr:123350K .......... .......... .......... .......... .......... 73% 42.4M 2s 2024-03-29T04:10:08.690 INFO:teuthology.orchestra.run.smithi156.stderr:123400K .......... .......... .......... .......... .......... 73% 42.7M 2s 2024-03-29T04:10:08.692 INFO:teuthology.orchestra.run.smithi156.stderr:123450K .......... .......... .......... .......... .......... 73% 41.9M 2s 2024-03-29T04:10:08.693 INFO:teuthology.orchestra.run.smithi156.stderr:123500K .......... .......... .......... .......... .......... 73% 36.6M 2s 2024-03-29T04:10:08.694 INFO:teuthology.orchestra.run.smithi156.stderr:123550K .......... .......... .......... .......... .......... 73% 42.1M 2s 2024-03-29T04:10:08.696 INFO:teuthology.orchestra.run.smithi156.stderr:123600K .......... .......... .......... .......... .......... 73% 40.8M 2s 2024-03-29T04:10:08.697 INFO:teuthology.orchestra.run.smithi156.stderr:123650K .......... .......... .......... .......... .......... 73% 40.9M 2s 2024-03-29T04:10:08.698 INFO:teuthology.orchestra.run.smithi156.stderr:123700K .......... .......... .......... .......... .......... 73% 35.8M 2s 2024-03-29T04:10:08.700 INFO:teuthology.orchestra.run.smithi156.stderr:123750K .......... .......... .......... .......... .......... 73% 40.7M 2s 2024-03-29T04:10:08.701 INFO:teuthology.orchestra.run.smithi156.stderr:123800K .......... .......... .......... .......... .......... 73% 38.9M 2s 2024-03-29T04:10:08.702 INFO:teuthology.orchestra.run.smithi156.stderr:123850K .......... .......... .......... .......... .......... 73% 41.4M 2s 2024-03-29T04:10:08.704 INFO:teuthology.orchestra.run.smithi156.stderr:123900K .......... .......... .......... .......... .......... 73% 25.7M 2s 2024-03-29T04:10:08.705 INFO:teuthology.orchestra.run.smithi156.stderr:123950K .......... .......... .......... .......... .......... 73% 28.2M 2s 2024-03-29T04:10:08.707 INFO:teuthology.orchestra.run.smithi156.stderr:124000K .......... .......... .......... .......... .......... 73% 27.4M 2s 2024-03-29T04:10:08.709 INFO:teuthology.orchestra.run.smithi156.stderr:124050K .......... .......... .......... .......... .......... 73% 28.2M 2s 2024-03-29T04:10:08.711 INFO:teuthology.orchestra.run.smithi156.stderr:124100K .......... .......... .......... .......... .......... 73% 23.7M 2s 2024-03-29T04:10:08.713 INFO:teuthology.orchestra.run.smithi156.stderr:124150K .......... .......... .......... .......... .......... 73% 31.4M 2s 2024-03-29T04:10:08.714 INFO:teuthology.orchestra.run.smithi156.stderr:124200K .......... .......... .......... .......... .......... 73% 32.3M 2s 2024-03-29T04:10:08.715 INFO:teuthology.orchestra.run.smithi156.stderr:124250K .......... .......... .......... .......... .......... 73% 35.5M 2s 2024-03-29T04:10:08.717 INFO:teuthology.orchestra.run.smithi156.stderr:124300K .......... .......... .......... .......... .......... 73% 30.9M 2s 2024-03-29T04:10:08.718 INFO:teuthology.orchestra.run.smithi156.stderr:124350K .......... .......... .......... .......... .......... 73% 36.7M 2s 2024-03-29T04:10:08.720 INFO:teuthology.orchestra.run.smithi156.stderr:124400K .......... .......... .......... .......... .......... 73% 36.2M 2s 2024-03-29T04:10:08.721 INFO:teuthology.orchestra.run.smithi156.stderr:124450K .......... .......... .......... .......... .......... 73% 35.4M 2s 2024-03-29T04:10:08.723 INFO:teuthology.orchestra.run.smithi156.stderr:124500K .......... .......... .......... .......... .......... 73% 31.7M 2s 2024-03-29T04:10:08.724 INFO:teuthology.orchestra.run.smithi156.stderr:124550K .......... .......... .......... .......... .......... 73% 30.9M 2s 2024-03-29T04:10:08.726 INFO:teuthology.orchestra.run.smithi156.stderr:124600K .......... .......... .......... .......... .......... 73% 30.1M 2s 2024-03-29T04:10:08.727 INFO:teuthology.orchestra.run.smithi156.stderr:124650K .......... .......... .......... .......... .......... 73% 33.8M 2s 2024-03-29T04:10:08.729 INFO:teuthology.orchestra.run.smithi156.stderr:124700K .......... .......... .......... .......... .......... 74% 31.8M 2s 2024-03-29T04:10:08.730 INFO:teuthology.orchestra.run.smithi156.stderr:124750K .......... .......... .......... .......... .......... 74% 36.4M 2s 2024-03-29T04:10:08.731 INFO:teuthology.orchestra.run.smithi156.stderr:124800K .......... .......... .......... .......... .......... 74% 36.0M 2s 2024-03-29T04:10:08.733 INFO:teuthology.orchestra.run.smithi156.stderr:124850K .......... .......... .......... .......... .......... 74% 35.4M 2s 2024-03-29T04:10:08.735 INFO:teuthology.orchestra.run.smithi156.stderr:124900K .......... .......... .......... .......... .......... 74% 30.2M 2s 2024-03-29T04:10:08.736 INFO:teuthology.orchestra.run.smithi156.stderr:124950K .......... .......... .......... .......... .......... 74% 35.6M 2s 2024-03-29T04:10:08.737 INFO:teuthology.orchestra.run.smithi156.stderr:125000K .......... .......... .......... .......... .......... 74% 35.0M 2s 2024-03-29T04:10:08.739 INFO:teuthology.orchestra.run.smithi156.stderr:125050K .......... .......... .......... .......... .......... 74% 35.4M 2s 2024-03-29T04:10:08.740 INFO:teuthology.orchestra.run.smithi156.stderr:125100K .......... .......... .......... .......... .......... 74% 30.6M 2s 2024-03-29T04:10:08.742 INFO:teuthology.orchestra.run.smithi156.stderr:125150K .......... .......... .......... .......... .......... 74% 35.1M 2s 2024-03-29T04:10:08.743 INFO:teuthology.orchestra.run.smithi156.stderr:125200K .......... .......... .......... .......... .......... 74% 36.0M 2s 2024-03-29T04:10:08.744 INFO:teuthology.orchestra.run.smithi156.stderr:125250K .......... .......... .......... .......... .......... 74% 34.4M 2s 2024-03-29T04:10:08.746 INFO:teuthology.orchestra.run.smithi156.stderr:125300K .......... .......... .......... .......... .......... 74% 29.9M 2s 2024-03-29T04:10:08.748 INFO:teuthology.orchestra.run.smithi156.stderr:125350K .......... .......... .......... .......... .......... 74% 30.4M 2s 2024-03-29T04:10:08.750 INFO:teuthology.orchestra.run.smithi156.stderr:125400K .......... .......... .......... .......... .......... 74% 23.3M 2s 2024-03-29T04:10:08.752 INFO:teuthology.orchestra.run.smithi156.stderr:125450K .......... .......... .......... .......... .......... 74% 28.6M 2s 2024-03-29T04:10:08.753 INFO:teuthology.orchestra.run.smithi156.stderr:125500K .......... .......... .......... .......... .......... 74% 24.1M 2s 2024-03-29T04:10:08.755 INFO:teuthology.orchestra.run.smithi156.stderr:125550K .......... .......... .......... .......... .......... 74% 28.0M 2s 2024-03-29T04:10:08.757 INFO:teuthology.orchestra.run.smithi156.stderr:125600K .......... .......... .......... .......... .......... 74% 27.7M 2s 2024-03-29T04:10:08.759 INFO:teuthology.orchestra.run.smithi156.stderr:125650K .......... .......... .......... .......... .......... 74% 28.2M 2s 2024-03-29T04:10:08.761 INFO:teuthology.orchestra.run.smithi156.stderr:125700K .......... .......... .......... .......... .......... 74% 21.6M 2s 2024-03-29T04:10:08.763 INFO:teuthology.orchestra.run.smithi156.stderr:125750K .......... .......... .......... .......... .......... 74% 24.8M 2s 2024-03-29T04:10:08.765 INFO:teuthology.orchestra.run.smithi156.stderr:125800K .......... .......... .......... .......... .......... 74% 24.4M 2s 2024-03-29T04:10:08.767 INFO:teuthology.orchestra.run.smithi156.stderr:125850K .......... .......... .......... .......... .......... 74% 24.7M 2s 2024-03-29T04:10:08.769 INFO:teuthology.orchestra.run.smithi156.stderr:125900K .......... .......... .......... .......... .......... 74% 20.4M 2s 2024-03-29T04:10:08.771 INFO:teuthology.orchestra.run.smithi156.stderr:125950K .......... .......... .......... .......... .......... 74% 24.3M 2s 2024-03-29T04:10:08.774 INFO:teuthology.orchestra.run.smithi156.stderr:126000K .......... .......... .......... .......... .......... 74% 24.1M 2s 2024-03-29T04:10:08.776 INFO:teuthology.orchestra.run.smithi156.stderr:126050K .......... .......... .......... .......... .......... 74% 23.9M 2s 2024-03-29T04:10:08.778 INFO:teuthology.orchestra.run.smithi156.stderr:126100K .......... .......... .......... .......... .......... 74% 21.4M 2s 2024-03-29T04:10:08.780 INFO:teuthology.orchestra.run.smithi156.stderr:126150K .......... .......... .......... .......... .......... 74% 24.6M 2s 2024-03-29T04:10:08.782 INFO:teuthology.orchestra.run.smithi156.stderr:126200K .......... .......... .......... .......... .......... 74% 25.3M 2s 2024-03-29T04:10:08.784 INFO:teuthology.orchestra.run.smithi156.stderr:126250K .......... .......... .......... .......... .......... 74% 26.7M 2s 2024-03-29T04:10:08.785 INFO:teuthology.orchestra.run.smithi156.stderr:126300K .......... .......... .......... .......... .......... 74% 23.9M 2s 2024-03-29T04:10:08.787 INFO:teuthology.orchestra.run.smithi156.stderr:126350K .......... .......... .......... .......... .......... 74% 27.5M 2s 2024-03-29T04:10:08.789 INFO:teuthology.orchestra.run.smithi156.stderr:126400K .......... .......... .......... .......... .......... 75% 27.5M 2s 2024-03-29T04:10:08.791 INFO:teuthology.orchestra.run.smithi156.stderr:126450K .......... .......... .......... .......... .......... 75% 27.6M 2s 2024-03-29T04:10:08.793 INFO:teuthology.orchestra.run.smithi156.stderr:126500K .......... .......... .......... .......... .......... 75% 23.3M 2s 2024-03-29T04:10:08.795 INFO:teuthology.orchestra.run.smithi156.stderr:126550K .......... .......... .......... .......... .......... 75% 27.3M 2s 2024-03-29T04:10:08.796 INFO:teuthology.orchestra.run.smithi156.stderr:126600K .......... .......... .......... .......... .......... 75% 26.6M 2s 2024-03-29T04:10:08.798 INFO:teuthology.orchestra.run.smithi156.stderr:126650K .......... .......... .......... .......... .......... 75% 27.3M 2s 2024-03-29T04:10:08.800 INFO:teuthology.orchestra.run.smithi156.stderr:126700K .......... .......... .......... .......... .......... 75% 23.2M 2s 2024-03-29T04:10:08.802 INFO:teuthology.orchestra.run.smithi156.stderr:126750K .......... .......... .......... .......... .......... 75% 27.3M 2s 2024-03-29T04:10:08.804 INFO:teuthology.orchestra.run.smithi156.stderr:126800K .......... .......... .......... .......... .......... 75% 27.0M 2s 2024-03-29T04:10:08.806 INFO:teuthology.orchestra.run.smithi156.stderr:126850K .......... .......... .......... .......... .......... 75% 25.5M 2s 2024-03-29T04:10:08.808 INFO:teuthology.orchestra.run.smithi156.stderr:126900K .......... .......... .......... .......... .......... 75% 20.0M 2s 2024-03-29T04:10:08.811 INFO:teuthology.orchestra.run.smithi156.stderr:126950K .......... .......... .......... .......... .......... 75% 22.0M 2s 2024-03-29T04:10:08.812 INFO:teuthology.orchestra.run.smithi156.stderr:127000K .......... .......... .......... .......... .......... 75% 24.7M 2s 2024-03-29T04:10:08.815 INFO:teuthology.orchestra.run.smithi156.stderr:127050K .......... .......... .......... .......... .......... 75% 29.2M 2s 2024-03-29T04:10:08.816 INFO:teuthology.orchestra.run.smithi156.stderr:127100K .......... .......... .......... .......... .......... 75% 31.3M 2s 2024-03-29T04:10:08.817 INFO:teuthology.orchestra.run.smithi156.stderr:127150K .......... .......... .......... .......... .......... 75% 37.2M 2s 2024-03-29T04:10:08.818 INFO:teuthology.orchestra.run.smithi156.stderr:127200K .......... .......... .......... .......... .......... 75% 39.6M 2s 2024-03-29T04:10:08.820 INFO:teuthology.orchestra.run.smithi156.stderr:127250K .......... .......... .......... .......... .......... 75% 40.8M 1s 2024-03-29T04:10:08.821 INFO:teuthology.orchestra.run.smithi156.stderr:127300K .......... .......... .......... .......... .......... 75% 25.8M 1s 2024-03-29T04:10:08.823 INFO:teuthology.orchestra.run.smithi156.stderr:127350K .......... .......... .......... .......... .......... 75% 34.1M 1s 2024-03-29T04:10:08.824 INFO:teuthology.orchestra.run.smithi156.stderr:127400K .......... .......... .......... .......... .......... 75% 35.8M 1s 2024-03-29T04:10:08.826 INFO:teuthology.orchestra.run.smithi156.stderr:127450K .......... .......... .......... .......... .......... 75% 37.2M 1s 2024-03-29T04:10:08.827 INFO:teuthology.orchestra.run.smithi156.stderr:127500K .......... .......... .......... .......... .......... 75% 26.3M 1s 2024-03-29T04:10:08.829 INFO:teuthology.orchestra.run.smithi156.stderr:127550K .......... .......... .......... .......... .......... 75% 35.8M 1s 2024-03-29T04:10:08.830 INFO:teuthology.orchestra.run.smithi156.stderr:127600K .......... .......... .......... .......... .......... 75% 35.3M 1s 2024-03-29T04:10:08.831 INFO:teuthology.orchestra.run.smithi156.stderr:127650K .......... .......... .......... .......... .......... 75% 36.0M 1s 2024-03-29T04:10:08.833 INFO:teuthology.orchestra.run.smithi156.stderr:127700K .......... .......... .......... .......... .......... 75% 31.2M 1s 2024-03-29T04:10:08.834 INFO:teuthology.orchestra.run.smithi156.stderr:127750K .......... .......... .......... .......... .......... 75% 37.3M 1s 2024-03-29T04:10:08.836 INFO:teuthology.orchestra.run.smithi156.stderr:127800K .......... .......... .......... .......... .......... 75% 36.3M 1s 2024-03-29T04:10:08.837 INFO:teuthology.orchestra.run.smithi156.stderr:127850K .......... .......... .......... .......... .......... 75% 35.9M 1s 2024-03-29T04:10:08.839 INFO:teuthology.orchestra.run.smithi156.stderr:127900K .......... .......... .......... .......... .......... 75% 30.8M 1s 2024-03-29T04:10:08.840 INFO:teuthology.orchestra.run.smithi156.stderr:127950K .......... .......... .......... .......... .......... 75% 35.9M 1s 2024-03-29T04:10:08.841 INFO:teuthology.orchestra.run.smithi156.stderr:128000K .......... .......... .......... .......... .......... 75% 36.4M 1s 2024-03-29T04:10:08.843 INFO:teuthology.orchestra.run.smithi156.stderr:128050K .......... .......... .......... .......... .......... 75% 35.5M 1s 2024-03-29T04:10:08.844 INFO:teuthology.orchestra.run.smithi156.stderr:128100K .......... .......... .......... .......... .......... 76% 30.3M 1s 2024-03-29T04:10:08.848 INFO:teuthology.orchestra.run.smithi156.stderr:128150K .......... .......... .......... .......... .......... 76% 36.1M 1s 2024-03-29T04:10:08.849 INFO:teuthology.orchestra.run.smithi156.stderr:128200K .......... .......... .......... .......... .......... 76% 35.3M 1s 2024-03-29T04:10:08.849 INFO:teuthology.orchestra.run.smithi156.stderr:128250K .......... .......... .......... .......... .......... 76% 38.9M 1s 2024-03-29T04:10:08.850 INFO:teuthology.orchestra.run.smithi156.stderr:128300K .......... .......... .......... .......... .......... 76% 33.5M 1s 2024-03-29T04:10:08.851 INFO:teuthology.orchestra.run.smithi156.stderr:128350K .......... .......... .......... .......... .......... 76% 38.7M 1s 2024-03-29T04:10:08.852 INFO:teuthology.orchestra.run.smithi156.stderr:128400K .......... .......... .......... .......... .......... 76% 37.4M 1s 2024-03-29T04:10:08.854 INFO:teuthology.orchestra.run.smithi156.stderr:128450K .......... .......... .......... .......... .......... 76% 39.4M 1s 2024-03-29T04:10:08.855 INFO:teuthology.orchestra.run.smithi156.stderr:128500K .......... .......... .......... .......... .......... 76% 33.3M 1s 2024-03-29T04:10:08.857 INFO:teuthology.orchestra.run.smithi156.stderr:128550K .......... .......... .......... .......... .......... 76% 36.2M 1s 2024-03-29T04:10:08.858 INFO:teuthology.orchestra.run.smithi156.stderr:128600K .......... .......... .......... .......... .......... 76% 37.4M 1s 2024-03-29T04:10:08.859 INFO:teuthology.orchestra.run.smithi156.stderr:128650K .......... .......... .......... .......... .......... 76% 34.2M 1s 2024-03-29T04:10:08.861 INFO:teuthology.orchestra.run.smithi156.stderr:128700K .......... .......... .......... .......... .......... 76% 30.0M 1s 2024-03-29T04:10:08.862 INFO:teuthology.orchestra.run.smithi156.stderr:128750K .......... .......... .......... .......... .......... 76% 35.0M 1s 2024-03-29T04:10:08.864 INFO:teuthology.orchestra.run.smithi156.stderr:128800K .......... .......... .......... .......... .......... 76% 35.4M 1s 2024-03-29T04:10:08.865 INFO:teuthology.orchestra.run.smithi156.stderr:128850K .......... .......... .......... .......... .......... 76% 34.1M 1s 2024-03-29T04:10:08.869 INFO:teuthology.orchestra.run.smithi156.stderr:128900K .......... .......... .......... .......... .......... 76% 29.3M 1s 2024-03-29T04:10:08.869 INFO:teuthology.orchestra.run.smithi156.stderr:128950K .......... .......... .......... .......... .......... 76% 33.8M 1s 2024-03-29T04:10:08.870 INFO:teuthology.orchestra.run.smithi156.stderr:129000K .......... .......... .......... .......... .......... 76% 34.2M 1s 2024-03-29T04:10:08.871 INFO:teuthology.orchestra.run.smithi156.stderr:129050K .......... .......... .......... .......... .......... 76% 33.7M 1s 2024-03-29T04:10:08.873 INFO:teuthology.orchestra.run.smithi156.stderr:129100K .......... .......... .......... .......... .......... 76% 29.3M 1s 2024-03-29T04:10:08.874 INFO:teuthology.orchestra.run.smithi156.stderr:129150K .......... .......... .......... .......... .......... 76% 33.9M 1s 2024-03-29T04:10:08.875 INFO:teuthology.orchestra.run.smithi156.stderr:129200K .......... .......... .......... .......... .......... 76% 33.5M 1s 2024-03-29T04:10:08.877 INFO:teuthology.orchestra.run.smithi156.stderr:129250K .......... .......... .......... .......... .......... 76% 33.2M 1s 2024-03-29T04:10:08.879 INFO:teuthology.orchestra.run.smithi156.stderr:129300K .......... .......... .......... .......... .......... 76% 29.0M 1s 2024-03-29T04:10:08.880 INFO:teuthology.orchestra.run.smithi156.stderr:129350K .......... .......... .......... .......... .......... 76% 33.2M 1s 2024-03-29T04:10:08.881 INFO:teuthology.orchestra.run.smithi156.stderr:129400K .......... .......... .......... .......... .......... 76% 34.0M 1s 2024-03-29T04:10:08.883 INFO:teuthology.orchestra.run.smithi156.stderr:129450K .......... .......... .......... .......... .......... 76% 33.1M 1s 2024-03-29T04:10:08.885 INFO:teuthology.orchestra.run.smithi156.stderr:129500K .......... .......... .......... .......... .......... 76% 27.9M 1s 2024-03-29T04:10:08.886 INFO:teuthology.orchestra.run.smithi156.stderr:129550K .......... .......... .......... .......... .......... 76% 32.6M 1s 2024-03-29T04:10:08.888 INFO:teuthology.orchestra.run.smithi156.stderr:129600K .......... .......... .......... .......... .......... 76% 33.0M 1s 2024-03-29T04:10:08.889 INFO:teuthology.orchestra.run.smithi156.stderr:129650K .......... .......... .......... .......... .......... 76% 32.9M 1s 2024-03-29T04:10:08.891 INFO:teuthology.orchestra.run.smithi156.stderr:129700K .......... .......... .......... .......... .......... 76% 28.4M 1s 2024-03-29T04:10:08.893 INFO:teuthology.orchestra.run.smithi156.stderr:129750K .......... .......... .......... .......... .......... 77% 32.8M 1s 2024-03-29T04:10:08.894 INFO:teuthology.orchestra.run.smithi156.stderr:129800K .......... .......... .......... .......... .......... 77% 32.2M 1s 2024-03-29T04:10:08.896 INFO:teuthology.orchestra.run.smithi156.stderr:129850K .......... .......... .......... .......... .......... 77% 32.0M 1s 2024-03-29T04:10:08.897 INFO:teuthology.orchestra.run.smithi156.stderr:129900K .......... .......... .......... .......... .......... 77% 27.1M 1s 2024-03-29T04:10:08.899 INFO:teuthology.orchestra.run.smithi156.stderr:129950K .......... .......... .......... .......... .......... 77% 31.3M 1s 2024-03-29T04:10:08.900 INFO:teuthology.orchestra.run.smithi156.stderr:130000K .......... .......... .......... .......... .......... 77% 32.0M 1s 2024-03-29T04:10:08.902 INFO:teuthology.orchestra.run.smithi156.stderr:130050K .......... .......... .......... .......... .......... 77% 29.9M 1s 2024-03-29T04:10:08.904 INFO:teuthology.orchestra.run.smithi156.stderr:130100K .......... .......... .......... .......... .......... 77% 26.1M 1s 2024-03-29T04:10:08.906 INFO:teuthology.orchestra.run.smithi156.stderr:130150K .......... .......... .......... .......... .......... 77% 30.4M 1s 2024-03-29T04:10:08.907 INFO:teuthology.orchestra.run.smithi156.stderr:130200K .......... .......... .......... .......... .......... 77% 30.6M 1s 2024-03-29T04:10:08.909 INFO:teuthology.orchestra.run.smithi156.stderr:130250K .......... .......... .......... .......... .......... 77% 31.3M 1s 2024-03-29T04:10:08.911 INFO:teuthology.orchestra.run.smithi156.stderr:130300K .......... .......... .......... .......... .......... 77% 26.1M 1s 2024-03-29T04:10:08.912 INFO:teuthology.orchestra.run.smithi156.stderr:130350K .......... .......... .......... .......... .......... 77% 30.5M 1s 2024-03-29T04:10:08.914 INFO:teuthology.orchestra.run.smithi156.stderr:130400K .......... .......... .......... .......... .......... 77% 29.1M 1s 2024-03-29T04:10:08.916 INFO:teuthology.orchestra.run.smithi156.stderr:130450K .......... .......... .......... .......... .......... 77% 29.6M 1s 2024-03-29T04:10:08.917 INFO:teuthology.orchestra.run.smithi156.stderr:130500K .......... .......... .......... .......... .......... 77% 25.3M 1s 2024-03-29T04:10:08.919 INFO:teuthology.orchestra.run.smithi156.stderr:130550K .......... .......... .......... .......... .......... 77% 29.2M 1s 2024-03-29T04:10:08.921 INFO:teuthology.orchestra.run.smithi156.stderr:130600K .......... .......... .......... .......... .......... 77% 29.3M 1s 2024-03-29T04:10:08.922 INFO:teuthology.orchestra.run.smithi156.stderr:130650K .......... .......... .......... .......... .......... 77% 29.1M 1s 2024-03-29T04:10:08.924 INFO:teuthology.orchestra.run.smithi156.stderr:130700K .......... .......... .......... .......... .......... 77% 25.2M 1s 2024-03-29T04:10:08.926 INFO:teuthology.orchestra.run.smithi156.stderr:130750K .......... .......... .......... .......... .......... 77% 28.7M 1s 2024-03-29T04:10:08.928 INFO:teuthology.orchestra.run.smithi156.stderr:130800K .......... .......... .......... .......... .......... 77% 25.8M 1s 2024-03-29T04:10:08.930 INFO:teuthology.orchestra.run.smithi156.stderr:130850K .......... .......... .......... .......... .......... 77% 29.9M 1s 2024-03-29T04:10:08.931 INFO:teuthology.orchestra.run.smithi156.stderr:130900K .......... .......... .......... .......... .......... 77% 27.1M 1s 2024-03-29T04:10:08.933 INFO:teuthology.orchestra.run.smithi156.stderr:130950K .......... .......... .......... .......... .......... 77% 32.3M 1s 2024-03-29T04:10:08.934 INFO:teuthology.orchestra.run.smithi156.stderr:131000K .......... .......... .......... .......... .......... 77% 32.5M 1s 2024-03-29T04:10:08.936 INFO:teuthology.orchestra.run.smithi156.stderr:131050K .......... .......... .......... .......... .......... 77% 29.5M 1s 2024-03-29T04:10:08.938 INFO:teuthology.orchestra.run.smithi156.stderr:131100K .......... .......... .......... .......... .......... 77% 25.7M 1s 2024-03-29T04:10:08.940 INFO:teuthology.orchestra.run.smithi156.stderr:131150K .......... .......... .......... .......... .......... 77% 28.1M 1s 2024-03-29T04:10:08.941 INFO:teuthology.orchestra.run.smithi156.stderr:131200K .......... .......... .......... .......... .......... 77% 26.6M 1s 2024-03-29T04:10:08.943 INFO:teuthology.orchestra.run.smithi156.stderr:131250K .......... .......... .......... .......... .......... 77% 30.9M 1s 2024-03-29T04:10:08.945 INFO:teuthology.orchestra.run.smithi156.stderr:131300K .......... .......... .......... .......... .......... 77% 25.1M 1s 2024-03-29T04:10:08.946 INFO:teuthology.orchestra.run.smithi156.stderr:131350K .......... .......... .......... .......... .......... 77% 37.2M 1s 2024-03-29T04:10:08.948 INFO:teuthology.orchestra.run.smithi156.stderr:131400K .......... .......... .......... .......... .......... 77% 28.5M 1s 2024-03-29T04:10:08.950 INFO:teuthology.orchestra.run.smithi156.stderr:131450K .......... .......... .......... .......... .......... 78% 34.4M 1s 2024-03-29T04:10:08.951 INFO:teuthology.orchestra.run.smithi156.stderr:131500K .......... .......... .......... .......... .......... 78% 28.9M 1s 2024-03-29T04:10:08.953 INFO:teuthology.orchestra.run.smithi156.stderr:131550K .......... .......... .......... .......... .......... 78% 34.6M 1s 2024-03-29T04:10:08.954 INFO:teuthology.orchestra.run.smithi156.stderr:131600K .......... .......... .......... .......... .......... 78% 34.4M 1s 2024-03-29T04:10:08.956 INFO:teuthology.orchestra.run.smithi156.stderr:131650K .......... .......... .......... .......... .......... 78% 34.3M 1s 2024-03-29T04:10:08.957 INFO:teuthology.orchestra.run.smithi156.stderr:131700K .......... .......... .......... .......... .......... 78% 31.0M 1s 2024-03-29T04:10:08.959 INFO:teuthology.orchestra.run.smithi156.stderr:131750K .......... .......... .......... .......... .......... 78% 34.0M 1s 2024-03-29T04:10:08.960 INFO:teuthology.orchestra.run.smithi156.stderr:131800K .......... .......... .......... .......... .......... 78% 34.5M 1s 2024-03-29T04:10:08.962 INFO:teuthology.orchestra.run.smithi156.stderr:131850K .......... .......... .......... .......... .......... 78% 35.4M 1s 2024-03-29T04:10:08.963 INFO:teuthology.orchestra.run.smithi156.stderr:131900K .......... .......... .......... .......... .......... 78% 28.5M 1s 2024-03-29T04:10:08.964 INFO:teuthology.orchestra.run.smithi156.stderr:131950K .......... .......... .......... .......... .......... 78% 32.4M 1s 2024-03-29T04:10:08.966 INFO:teuthology.orchestra.run.smithi156.stderr:132000K .......... .......... .......... .......... .......... 78% 32.5M 1s 2024-03-29T04:10:08.968 INFO:teuthology.orchestra.run.smithi156.stderr:132050K .......... .......... .......... .......... .......... 78% 32.8M 1s 2024-03-29T04:10:08.969 INFO:teuthology.orchestra.run.smithi156.stderr:132100K .......... .......... .......... .......... .......... 78% 27.6M 1s 2024-03-29T04:10:08.971 INFO:teuthology.orchestra.run.smithi156.stderr:132150K .......... .......... .......... .......... .......... 78% 33.4M 1s 2024-03-29T04:10:08.972 INFO:teuthology.orchestra.run.smithi156.stderr:132200K .......... .......... .......... .......... .......... 78% 32.9M 1s 2024-03-29T04:10:08.974 INFO:teuthology.orchestra.run.smithi156.stderr:132250K .......... .......... .......... .......... .......... 78% 33.1M 1s 2024-03-29T04:10:08.975 INFO:teuthology.orchestra.run.smithi156.stderr:132300K .......... .......... .......... .......... .......... 78% 28.0M 1s 2024-03-29T04:10:08.978 INFO:teuthology.orchestra.run.smithi156.stderr:132350K .......... .......... .......... .......... .......... 78% 33.1M 1s 2024-03-29T04:10:08.978 INFO:teuthology.orchestra.run.smithi156.stderr:132400K .......... .......... .......... .......... .......... 78% 28.9M 1s 2024-03-29T04:10:08.980 INFO:teuthology.orchestra.run.smithi156.stderr:132450K .......... .......... .......... .......... .......... 78% 29.2M 1s 2024-03-29T04:10:08.982 INFO:teuthology.orchestra.run.smithi156.stderr:132500K .......... .......... .......... .......... .......... 78% 26.5M 1s 2024-03-29T04:10:08.984 INFO:teuthology.orchestra.run.smithi156.stderr:132550K .......... .......... .......... .......... .......... 78% 30.6M 1s 2024-03-29T04:10:08.985 INFO:teuthology.orchestra.run.smithi156.stderr:132600K .......... .......... .......... .......... .......... 78% 30.0M 1s 2024-03-29T04:10:08.987 INFO:teuthology.orchestra.run.smithi156.stderr:132650K .......... .......... .......... .......... .......... 78% 30.1M 1s 2024-03-29T04:10:08.989 INFO:teuthology.orchestra.run.smithi156.stderr:132700K .......... .......... .......... .......... .......... 78% 26.8M 1s 2024-03-29T04:10:08.990 INFO:teuthology.orchestra.run.smithi156.stderr:132750K .......... .......... .......... .......... .......... 78% 29.8M 1s 2024-03-29T04:10:08.992 INFO:teuthology.orchestra.run.smithi156.stderr:132800K .......... .......... .......... .......... .......... 78% 29.7M 1s 2024-03-29T04:10:08.994 INFO:teuthology.orchestra.run.smithi156.stderr:132850K .......... .......... .......... .......... .......... 78% 30.3M 1s 2024-03-29T04:10:08.996 INFO:teuthology.orchestra.run.smithi156.stderr:132900K .......... .......... .......... .......... .......... 78% 18.9M 1s 2024-03-29T04:10:08.998 INFO:teuthology.orchestra.run.smithi156.stderr:132950K .......... .......... .......... .......... .......... 78% 25.9M 1s 2024-03-29T04:10:09.000 INFO:teuthology.orchestra.run.smithi156.stderr:133000K .......... .......... .......... .......... .......... 78% 26.8M 1s 2024-03-29T04:10:09.002 INFO:teuthology.orchestra.run.smithi156.stderr:133050K .......... .......... .......... .......... .......... 78% 23.9M 1s 2024-03-29T04:10:09.004 INFO:teuthology.orchestra.run.smithi156.stderr:133100K .......... .......... .......... .......... .......... 78% 22.4M 1s 2024-03-29T04:10:09.006 INFO:teuthology.orchestra.run.smithi156.stderr:133150K .......... .......... .......... .......... .......... 79% 33.7M 1s 2024-03-29T04:10:09.007 INFO:teuthology.orchestra.run.smithi156.stderr:133200K .......... .......... .......... .......... .......... 79% 34.2M 1s 2024-03-29T04:10:09.008 INFO:teuthology.orchestra.run.smithi156.stderr:133250K .......... .......... .......... .......... .......... 79% 35.4M 1s 2024-03-29T04:10:09.010 INFO:teuthology.orchestra.run.smithi156.stderr:133300K .......... .......... .......... .......... .......... 79% 30.6M 1s 2024-03-29T04:10:09.012 INFO:teuthology.orchestra.run.smithi156.stderr:133350K .......... .......... .......... .......... .......... 79% 32.3M 1s 2024-03-29T04:10:09.013 INFO:teuthology.orchestra.run.smithi156.stderr:133400K .......... .......... .......... .......... .......... 79% 32.9M 1s 2024-03-29T04:10:09.015 INFO:teuthology.orchestra.run.smithi156.stderr:133450K .......... .......... .......... .......... .......... 79% 33.0M 1s 2024-03-29T04:10:09.016 INFO:teuthology.orchestra.run.smithi156.stderr:133500K .......... .......... .......... .......... .......... 79% 28.7M 1s 2024-03-29T04:10:09.018 INFO:teuthology.orchestra.run.smithi156.stderr:133550K .......... .......... .......... .......... .......... 79% 29.5M 1s 2024-03-29T04:10:09.019 INFO:teuthology.orchestra.run.smithi156.stderr:133600K .......... .......... .......... .......... .......... 79% 30.3M 1s 2024-03-29T04:10:09.021 INFO:teuthology.orchestra.run.smithi156.stderr:133650K .......... .......... .......... .......... .......... 79% 29.1M 1s 2024-03-29T04:10:09.023 INFO:teuthology.orchestra.run.smithi156.stderr:133700K .......... .......... .......... .......... .......... 79% 26.4M 1s 2024-03-29T04:10:09.025 INFO:teuthology.orchestra.run.smithi156.stderr:133750K .......... .......... .......... .......... .......... 79% 28.1M 1s 2024-03-29T04:10:09.026 INFO:teuthology.orchestra.run.smithi156.stderr:133800K .......... .......... .......... .......... .......... 79% 27.8M 1s 2024-03-29T04:10:09.028 INFO:teuthology.orchestra.run.smithi156.stderr:133850K .......... .......... .......... .......... .......... 79% 28.1M 1s 2024-03-29T04:10:09.030 INFO:teuthology.orchestra.run.smithi156.stderr:133900K .......... .......... .......... .......... .......... 79% 24.6M 1s 2024-03-29T04:10:09.032 INFO:teuthology.orchestra.run.smithi156.stderr:133950K .......... .......... .......... .......... .......... 79% 27.9M 1s 2024-03-29T04:10:09.034 INFO:teuthology.orchestra.run.smithi156.stderr:134000K .......... .......... .......... .......... .......... 79% 28.5M 1s 2024-03-29T04:10:09.036 INFO:teuthology.orchestra.run.smithi156.stderr:134050K .......... .......... .......... .......... .......... 79% 27.7M 1s 2024-03-29T04:10:09.038 INFO:teuthology.orchestra.run.smithi156.stderr:134100K .......... .......... .......... .......... .......... 79% 23.7M 1s 2024-03-29T04:10:09.039 INFO:teuthology.orchestra.run.smithi156.stderr:134150K .......... .......... .......... .......... .......... 79% 27.5M 1s 2024-03-29T04:10:09.041 INFO:teuthology.orchestra.run.smithi156.stderr:134200K .......... .......... .......... .......... .......... 79% 28.0M 1s 2024-03-29T04:10:09.043 INFO:teuthology.orchestra.run.smithi156.stderr:134250K .......... .......... .......... .......... .......... 79% 24.1M 1s 2024-03-29T04:10:09.045 INFO:teuthology.orchestra.run.smithi156.stderr:134300K .......... .......... .......... .......... .......... 79% 24.0M 1s 2024-03-29T04:10:09.047 INFO:teuthology.orchestra.run.smithi156.stderr:134350K .......... .......... .......... .......... .......... 79% 27.4M 1s 2024-03-29T04:10:09.048 INFO:teuthology.orchestra.run.smithi156.stderr:134400K .......... .......... .......... .......... .......... 79% 33.3M 1s 2024-03-29T04:10:09.050 INFO:teuthology.orchestra.run.smithi156.stderr:134450K .......... .......... .......... .......... .......... 79% 33.7M 1s 2024-03-29T04:10:09.052 INFO:teuthology.orchestra.run.smithi156.stderr:134500K .......... .......... .......... .......... .......... 79% 26.2M 1s 2024-03-29T04:10:09.053 INFO:teuthology.orchestra.run.smithi156.stderr:134550K .......... .......... .......... .......... .......... 79% 30.8M 1s 2024-03-29T04:10:09.055 INFO:teuthology.orchestra.run.smithi156.stderr:134600K .......... .......... .......... .......... .......... 79% 30.8M 1s 2024-03-29T04:10:09.056 INFO:teuthology.orchestra.run.smithi156.stderr:134650K .......... .......... .......... .......... .......... 79% 30.9M 1s 2024-03-29T04:10:09.058 INFO:teuthology.orchestra.run.smithi156.stderr:134700K .......... .......... .......... .......... .......... 79% 27.1M 1s 2024-03-29T04:10:09.060 INFO:teuthology.orchestra.run.smithi156.stderr:134750K .......... .......... .......... .......... .......... 79% 31.7M 1s 2024-03-29T04:10:09.061 INFO:teuthology.orchestra.run.smithi156.stderr:134800K .......... .......... .......... .......... .......... 79% 31.0M 1s 2024-03-29T04:10:09.063 INFO:teuthology.orchestra.run.smithi156.stderr:134850K .......... .......... .......... .......... .......... 80% 29.2M 1s 2024-03-29T04:10:09.065 INFO:teuthology.orchestra.run.smithi156.stderr:134900K .......... .......... .......... .......... .......... 80% 25.1M 1s 2024-03-29T04:10:09.067 INFO:teuthology.orchestra.run.smithi156.stderr:134950K .......... .......... .......... .......... .......... 80% 29.2M 1s 2024-03-29T04:10:09.068 INFO:teuthology.orchestra.run.smithi156.stderr:135000K .......... .......... .......... .......... .......... 80% 29.5M 1s 2024-03-29T04:10:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:135050K .......... .......... .......... .......... .......... 80% 28.1M 1s 2024-03-29T04:10:09.072 INFO:teuthology.orchestra.run.smithi156.stderr:135100K .......... .......... .......... .......... .......... 80% 22.7M 1s 2024-03-29T04:10:09.074 INFO:teuthology.orchestra.run.smithi156.stderr:135150K .......... .......... .......... .......... .......... 80% 24.3M 1s 2024-03-29T04:10:09.076 INFO:teuthology.orchestra.run.smithi156.stderr:135200K .......... .......... .......... .......... .......... 80% 34.7M 1s 2024-03-29T04:10:09.077 INFO:teuthology.orchestra.run.smithi156.stderr:135250K .......... .......... .......... .......... .......... 80% 31.7M 1s 2024-03-29T04:10:09.079 INFO:teuthology.orchestra.run.smithi156.stderr:135300K .......... .......... .......... .......... .......... 80% 25.7M 1s 2024-03-29T04:10:09.080 INFO:teuthology.orchestra.run.smithi156.stderr:135350K .......... .......... .......... .......... .......... 80% 39.9M 1s 2024-03-29T04:10:09.081 INFO:teuthology.orchestra.run.smithi156.stderr:135400K .......... .......... .......... .......... .......... 80% 39.4M 1s 2024-03-29T04:10:09.083 INFO:teuthology.orchestra.run.smithi156.stderr:135450K .......... .......... .......... .......... .......... 80% 40.1M 1s 2024-03-29T04:10:09.084 INFO:teuthology.orchestra.run.smithi156.stderr:135500K .......... .......... .......... .......... .......... 80% 32.7M 1s 2024-03-29T04:10:09.085 INFO:teuthology.orchestra.run.smithi156.stderr:135550K .......... .......... .......... .......... .......... 80% 39.4M 1s 2024-03-29T04:10:09.086 INFO:teuthology.orchestra.run.smithi156.stderr:135600K .......... .......... .......... .......... .......... 80% 39.8M 1s 2024-03-29T04:10:09.088 INFO:teuthology.orchestra.run.smithi156.stderr:135650K .......... .......... .......... .......... .......... 80% 38.2M 1s 2024-03-29T04:10:09.089 INFO:teuthology.orchestra.run.smithi156.stderr:135700K .......... .......... .......... .......... .......... 80% 34.1M 1s 2024-03-29T04:10:09.091 INFO:teuthology.orchestra.run.smithi156.stderr:135750K .......... .......... .......... .......... .......... 80% 37.5M 1s 2024-03-29T04:10:09.092 INFO:teuthology.orchestra.run.smithi156.stderr:135800K .......... .......... .......... .......... .......... 80% 39.1M 1s 2024-03-29T04:10:09.093 INFO:teuthology.orchestra.run.smithi156.stderr:135850K .......... .......... .......... .......... .......... 80% 35.8M 1s 2024-03-29T04:10:09.095 INFO:teuthology.orchestra.run.smithi156.stderr:135900K .......... .......... .......... .......... .......... 80% 31.9M 1s 2024-03-29T04:10:09.096 INFO:teuthology.orchestra.run.smithi156.stderr:135950K .......... .......... .......... .......... .......... 80% 36.5M 1s 2024-03-29T04:10:09.097 INFO:teuthology.orchestra.run.smithi156.stderr:136000K .......... .......... .......... .......... .......... 80% 39.1M 1s 2024-03-29T04:10:09.099 INFO:teuthology.orchestra.run.smithi156.stderr:136050K .......... .......... .......... .......... .......... 80% 36.8M 1s 2024-03-29T04:10:09.100 INFO:teuthology.orchestra.run.smithi156.stderr:136100K .......... .......... .......... .......... .......... 80% 32.7M 1s 2024-03-29T04:10:09.101 INFO:teuthology.orchestra.run.smithi156.stderr:136150K .......... .......... .......... .......... .......... 80% 38.1M 1s 2024-03-29T04:10:09.103 INFO:teuthology.orchestra.run.smithi156.stderr:136200K .......... .......... .......... .......... .......... 80% 35.0M 1s 2024-03-29T04:10:09.104 INFO:teuthology.orchestra.run.smithi156.stderr:136250K .......... .......... .......... .......... .......... 80% 36.9M 1s 2024-03-29T04:10:09.106 INFO:teuthology.orchestra.run.smithi156.stderr:136300K .......... .......... .......... .......... .......... 80% 32.5M 1s 2024-03-29T04:10:09.107 INFO:teuthology.orchestra.run.smithi156.stderr:136350K .......... .......... .......... .......... .......... 80% 31.9M 1s 2024-03-29T04:10:09.109 INFO:teuthology.orchestra.run.smithi156.stderr:136400K .......... .......... .......... .......... .......... 80% 31.9M 1s 2024-03-29T04:10:09.110 INFO:teuthology.orchestra.run.smithi156.stderr:136450K .......... .......... .......... .......... .......... 80% 31.9M 1s 2024-03-29T04:10:09.112 INFO:teuthology.orchestra.run.smithi156.stderr:136500K .......... .......... .......... .......... .......... 81% 27.2M 1s 2024-03-29T04:10:09.113 INFO:teuthology.orchestra.run.smithi156.stderr:136550K .......... .......... .......... .......... .......... 81% 31.8M 1s 2024-03-29T04:10:09.115 INFO:teuthology.orchestra.run.smithi156.stderr:136600K .......... .......... .......... .......... .......... 81% 31.3M 1s 2024-03-29T04:10:09.117 INFO:teuthology.orchestra.run.smithi156.stderr:136650K .......... .......... .......... .......... .......... 81% 32.0M 1s 2024-03-29T04:10:09.118 INFO:teuthology.orchestra.run.smithi156.stderr:136700K .......... .......... .......... .......... .......... 81% 26.8M 1s 2024-03-29T04:10:09.120 INFO:teuthology.orchestra.run.smithi156.stderr:136750K .......... .......... .......... .......... .......... 81% 30.8M 1s 2024-03-29T04:10:09.121 INFO:teuthology.orchestra.run.smithi156.stderr:136800K .......... .......... .......... .......... .......... 81% 31.2M 1s 2024-03-29T04:10:09.124 INFO:teuthology.orchestra.run.smithi156.stderr:136850K .......... .......... .......... .......... .......... 81% 23.5M 1s 2024-03-29T04:10:09.126 INFO:teuthology.orchestra.run.smithi156.stderr:136900K .......... .......... .......... .......... .......... 81% 23.3M 1s 2024-03-29T04:10:09.127 INFO:teuthology.orchestra.run.smithi156.stderr:136950K .......... .......... .......... .......... .......... 81% 27.7M 1s 2024-03-29T04:10:09.129 INFO:teuthology.orchestra.run.smithi156.stderr:137000K .......... .......... .......... .......... .......... 81% 27.4M 1s 2024-03-29T04:10:09.131 INFO:teuthology.orchestra.run.smithi156.stderr:137050K .......... .......... .......... .......... .......... 81% 27.7M 1s 2024-03-29T04:10:09.133 INFO:teuthology.orchestra.run.smithi156.stderr:137100K .......... .......... .......... .......... .......... 81% 23.5M 1s 2024-03-29T04:10:09.135 INFO:teuthology.orchestra.run.smithi156.stderr:137150K .......... .......... .......... .......... .......... 81% 27.3M 1s 2024-03-29T04:10:09.137 INFO:teuthology.orchestra.run.smithi156.stderr:137200K .......... .......... .......... .......... .......... 81% 29.4M 1s 2024-03-29T04:10:09.138 INFO:teuthology.orchestra.run.smithi156.stderr:137250K .......... .......... .......... .......... .......... 81% 30.8M 1s 2024-03-29T04:10:09.140 INFO:teuthology.orchestra.run.smithi156.stderr:137300K .......... .......... .......... .......... .......... 81% 26.5M 1s 2024-03-29T04:10:09.142 INFO:teuthology.orchestra.run.smithi156.stderr:137350K .......... .......... .......... .......... .......... 81% 30.1M 1s 2024-03-29T04:10:09.143 INFO:teuthology.orchestra.run.smithi156.stderr:137400K .......... .......... .......... .......... .......... 81% 30.7M 1s 2024-03-29T04:10:09.145 INFO:teuthology.orchestra.run.smithi156.stderr:137450K .......... .......... .......... .......... .......... 81% 29.5M 1s 2024-03-29T04:10:09.147 INFO:teuthology.orchestra.run.smithi156.stderr:137500K .......... .......... .......... .......... .......... 81% 25.2M 1s 2024-03-29T04:10:09.149 INFO:teuthology.orchestra.run.smithi156.stderr:137550K .......... .......... .......... .......... .......... 81% 30.0M 1s 2024-03-29T04:10:09.150 INFO:teuthology.orchestra.run.smithi156.stderr:137600K .......... .......... .......... .......... .......... 81% 29.8M 1s 2024-03-29T04:10:09.152 INFO:teuthology.orchestra.run.smithi156.stderr:137650K .......... .......... .......... .......... .......... 81% 30.1M 1s 2024-03-29T04:10:09.154 INFO:teuthology.orchestra.run.smithi156.stderr:137700K .......... .......... .......... .......... .......... 81% 24.9M 1s 2024-03-29T04:10:09.156 INFO:teuthology.orchestra.run.smithi156.stderr:137750K .......... .......... .......... .......... .......... 81% 28.4M 1s 2024-03-29T04:10:09.157 INFO:teuthology.orchestra.run.smithi156.stderr:137800K .......... .......... .......... .......... .......... 81% 29.6M 1s 2024-03-29T04:10:09.159 INFO:teuthology.orchestra.run.smithi156.stderr:137850K .......... .......... .......... .......... .......... 81% 29.1M 1s 2024-03-29T04:10:09.161 INFO:teuthology.orchestra.run.smithi156.stderr:137900K .......... .......... .......... .......... .......... 81% 25.0M 1s 2024-03-29T04:10:09.162 INFO:teuthology.orchestra.run.smithi156.stderr:137950K .......... .......... .......... .......... .......... 81% 28.7M 1s 2024-03-29T04:10:09.164 INFO:teuthology.orchestra.run.smithi156.stderr:138000K .......... .......... .......... .......... .......... 81% 27.9M 1s 2024-03-29T04:10:09.166 INFO:teuthology.orchestra.run.smithi156.stderr:138050K .......... .......... .......... .......... .......... 81% 28.4M 1s 2024-03-29T04:10:09.168 INFO:teuthology.orchestra.run.smithi156.stderr:138100K .......... .......... .......... .......... .......... 81% 24.5M 1s 2024-03-29T04:10:09.170 INFO:teuthology.orchestra.run.smithi156.stderr:138150K .......... .......... .......... .......... .......... 81% 28.0M 1s 2024-03-29T04:10:09.171 INFO:teuthology.orchestra.run.smithi156.stderr:138200K .......... .......... .......... .......... .......... 82% 28.0M 1s 2024-03-29T04:10:09.173 INFO:teuthology.orchestra.run.smithi156.stderr:138250K .......... .......... .......... .......... .......... 82% 27.7M 1s 2024-03-29T04:10:09.175 INFO:teuthology.orchestra.run.smithi156.stderr:138300K .......... .......... .......... .......... .......... 82% 25.1M 1s 2024-03-29T04:10:09.177 INFO:teuthology.orchestra.run.smithi156.stderr:138350K .......... .......... .......... .......... .......... 82% 29.4M 1s 2024-03-29T04:10:09.179 INFO:teuthology.orchestra.run.smithi156.stderr:138400K .......... .......... .......... .......... .......... 82% 25.3M 1s 2024-03-29T04:10:09.181 INFO:teuthology.orchestra.run.smithi156.stderr:138450K .......... .......... .......... .......... .......... 82% 23.7M 1s 2024-03-29T04:10:09.183 INFO:teuthology.orchestra.run.smithi156.stderr:138500K .......... .......... .......... .......... .......... 82% 20.7M 1s 2024-03-29T04:10:09.185 INFO:teuthology.orchestra.run.smithi156.stderr:138550K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-03-29T04:10:09.187 INFO:teuthology.orchestra.run.smithi156.stderr:138600K .......... .......... .......... .......... .......... 82% 22.7M 1s 2024-03-29T04:10:09.190 INFO:teuthology.orchestra.run.smithi156.stderr:138650K .......... .......... .......... .......... .......... 82% 22.5M 1s 2024-03-29T04:10:09.192 INFO:teuthology.orchestra.run.smithi156.stderr:138700K .......... .......... .......... .......... .......... 82% 19.3M 1s 2024-03-29T04:10:09.194 INFO:teuthology.orchestra.run.smithi156.stderr:138750K .......... .......... .......... .......... .......... 82% 21.6M 1s 2024-03-29T04:10:09.196 INFO:teuthology.orchestra.run.smithi156.stderr:138800K .......... .......... .......... .......... .......... 82% 22.1M 1s 2024-03-29T04:10:09.199 INFO:teuthology.orchestra.run.smithi156.stderr:138850K .......... .......... .......... .......... .......... 82% 23.2M 1s 2024-03-29T04:10:09.201 INFO:teuthology.orchestra.run.smithi156.stderr:138900K .......... .......... .......... .......... .......... 82% 19.2M 1s 2024-03-29T04:10:09.204 INFO:teuthology.orchestra.run.smithi156.stderr:138950K .......... .......... .......... .......... .......... 82% 22.6M 1s 2024-03-29T04:10:09.205 INFO:teuthology.orchestra.run.smithi156.stderr:139000K .......... .......... .......... .......... .......... 82% 22.7M 1s 2024-03-29T04:10:09.208 INFO:teuthology.orchestra.run.smithi156.stderr:139050K .......... .......... .......... .......... .......... 82% 22.6M 1s 2024-03-29T04:10:09.210 INFO:teuthology.orchestra.run.smithi156.stderr:139100K .......... .......... .......... .......... .......... 82% 19.4M 1s 2024-03-29T04:10:09.212 INFO:teuthology.orchestra.run.smithi156.stderr:139150K .......... .......... .......... .......... .......... 82% 23.1M 1s 2024-03-29T04:10:09.214 INFO:teuthology.orchestra.run.smithi156.stderr:139200K .......... .......... .......... .......... .......... 82% 22.9M 1s 2024-03-29T04:10:09.217 INFO:teuthology.orchestra.run.smithi156.stderr:139250K .......... .......... .......... .......... .......... 82% 22.7M 1s 2024-03-29T04:10:09.220 INFO:teuthology.orchestra.run.smithi156.stderr:139300K .......... .......... .......... .......... .......... 82% 22.5M 1s 2024-03-29T04:10:09.220 INFO:teuthology.orchestra.run.smithi156.stderr:139350K .......... .......... .......... .......... .......... 82% 31.1M 1s 2024-03-29T04:10:09.222 INFO:teuthology.orchestra.run.smithi156.stderr:139400K .......... .......... .......... .......... .......... 82% 33.9M 1s 2024-03-29T04:10:09.223 INFO:teuthology.orchestra.run.smithi156.stderr:139450K .......... .......... .......... .......... .......... 82% 35.3M 1s 2024-03-29T04:10:09.225 INFO:teuthology.orchestra.run.smithi156.stderr:139500K .......... .......... .......... .......... .......... 82% 30.7M 1s 2024-03-29T04:10:09.226 INFO:teuthology.orchestra.run.smithi156.stderr:139550K .......... .......... .......... .......... .......... 82% 36.3M 1s 2024-03-29T04:10:09.227 INFO:teuthology.orchestra.run.smithi156.stderr:139600K .......... .......... .......... .......... .......... 82% 36.7M 1s 2024-03-29T04:10:09.229 INFO:teuthology.orchestra.run.smithi156.stderr:139650K .......... .......... .......... .......... .......... 82% 28.5M 1s 2024-03-29T04:10:09.231 INFO:teuthology.orchestra.run.smithi156.stderr:139700K .......... .......... .......... .......... .......... 82% 24.8M 1s 2024-03-29T04:10:09.233 INFO:teuthology.orchestra.run.smithi156.stderr:139750K .......... .......... .......... .......... .......... 82% 29.4M 1s 2024-03-29T04:10:09.234 INFO:teuthology.orchestra.run.smithi156.stderr:139800K .......... .......... .......... .......... .......... 82% 28.6M 1s 2024-03-29T04:10:09.236 INFO:teuthology.orchestra.run.smithi156.stderr:139850K .......... .......... .......... .......... .......... 82% 30.5M 1s 2024-03-29T04:10:09.238 INFO:teuthology.orchestra.run.smithi156.stderr:139900K .......... .......... .......... .......... .......... 83% 29.9M 1s 2024-03-29T04:10:09.239 INFO:teuthology.orchestra.run.smithi156.stderr:139950K .......... .......... .......... .......... .......... 83% 34.4M 1s 2024-03-29T04:10:09.240 INFO:teuthology.orchestra.run.smithi156.stderr:140000K .......... .......... .......... .......... .......... 83% 35.1M 1s 2024-03-29T04:10:09.242 INFO:teuthology.orchestra.run.smithi156.stderr:140050K .......... .......... .......... .......... .......... 83% 35.1M 1s 2024-03-29T04:10:09.243 INFO:teuthology.orchestra.run.smithi156.stderr:140100K .......... .......... .......... .......... .......... 83% 31.3M 1s 2024-03-29T04:10:09.245 INFO:teuthology.orchestra.run.smithi156.stderr:140150K .......... .......... .......... .......... .......... 83% 29.3M 1s 2024-03-29T04:10:09.247 INFO:teuthology.orchestra.run.smithi156.stderr:140200K .......... .......... .......... .......... .......... 83% 30.3M 1s 2024-03-29T04:10:09.249 INFO:teuthology.orchestra.run.smithi156.stderr:140250K .......... .......... .......... .......... .......... 83% 28.1M 1s 2024-03-29T04:10:09.251 INFO:teuthology.orchestra.run.smithi156.stderr:140300K .......... .......... .......... .......... .......... 83% 25.0M 1s 2024-03-29T04:10:09.252 INFO:teuthology.orchestra.run.smithi156.stderr:140350K .......... .......... .......... .......... .......... 83% 29.2M 1s 2024-03-29T04:10:09.254 INFO:teuthology.orchestra.run.smithi156.stderr:140400K .......... .......... .......... .......... .......... 83% 29.3M 1s 2024-03-29T04:10:09.256 INFO:teuthology.orchestra.run.smithi156.stderr:140450K .......... .......... .......... .......... .......... 83% 28.8M 1s 2024-03-29T04:10:09.257 INFO:teuthology.orchestra.run.smithi156.stderr:140500K .......... .......... .......... .......... .......... 83% 30.4M 1s 2024-03-29T04:10:09.259 INFO:teuthology.orchestra.run.smithi156.stderr:140550K .......... .......... .......... .......... .......... 83% 37.7M 1s 2024-03-29T04:10:09.260 INFO:teuthology.orchestra.run.smithi156.stderr:140600K .......... .......... .......... .......... .......... 83% 37.7M 1s 2024-03-29T04:10:09.261 INFO:teuthology.orchestra.run.smithi156.stderr:140650K .......... .......... .......... .......... .......... 83% 37.4M 1s 2024-03-29T04:10:09.262 INFO:teuthology.orchestra.run.smithi156.stderr:140700K .......... .......... .......... .......... .......... 83% 32.3M 1s 2024-03-29T04:10:09.264 INFO:teuthology.orchestra.run.smithi156.stderr:140750K .......... .......... .......... .......... .......... 83% 36.6M 1s 2024-03-29T04:10:09.265 INFO:teuthology.orchestra.run.smithi156.stderr:140800K .......... .......... .......... .......... .......... 83% 36.9M 1s 2024-03-29T04:10:09.267 INFO:teuthology.orchestra.run.smithi156.stderr:140850K .......... .......... .......... .......... .......... 83% 37.0M 1s 2024-03-29T04:10:09.268 INFO:teuthology.orchestra.run.smithi156.stderr:140900K .......... .......... .......... .......... .......... 83% 31.4M 1s 2024-03-29T04:10:09.269 INFO:teuthology.orchestra.run.smithi156.stderr:140950K .......... .......... .......... .......... .......... 83% 35.6M 1s 2024-03-29T04:10:09.271 INFO:teuthology.orchestra.run.smithi156.stderr:141000K .......... .......... .......... .......... .......... 83% 35.8M 1s 2024-03-29T04:10:09.272 INFO:teuthology.orchestra.run.smithi156.stderr:141050K .......... .......... .......... .......... .......... 83% 35.2M 1s 2024-03-29T04:10:09.274 INFO:teuthology.orchestra.run.smithi156.stderr:141100K .......... .......... .......... .......... .......... 83% 30.5M 1s 2024-03-29T04:10:09.275 INFO:teuthology.orchestra.run.smithi156.stderr:141150K .......... .......... .......... .......... .......... 83% 35.0M 1s 2024-03-29T04:10:09.277 INFO:teuthology.orchestra.run.smithi156.stderr:141200K .......... .......... .......... .......... .......... 83% 34.7M 1s 2024-03-29T04:10:09.278 INFO:teuthology.orchestra.run.smithi156.stderr:141250K .......... .......... .......... .......... .......... 83% 34.7M 1s 2024-03-29T04:10:09.280 INFO:teuthology.orchestra.run.smithi156.stderr:141300K .......... .......... .......... .......... .......... 83% 24.5M 1s 2024-03-29T04:10:09.281 INFO:teuthology.orchestra.run.smithi156.stderr:141350K .......... .......... .......... .......... .......... 83% 30.9M 1s 2024-03-29T04:10:09.283 INFO:teuthology.orchestra.run.smithi156.stderr:141400K .......... .......... .......... .......... .......... 83% 34.8M 1s 2024-03-29T04:10:09.284 INFO:teuthology.orchestra.run.smithi156.stderr:141450K .......... .......... .......... .......... .......... 83% 33.9M 1s 2024-03-29T04:10:09.286 INFO:teuthology.orchestra.run.smithi156.stderr:141500K .......... .......... .......... .......... .......... 83% 29.2M 1s 2024-03-29T04:10:09.287 INFO:teuthology.orchestra.run.smithi156.stderr:141550K .......... .......... .......... .......... .......... 84% 34.0M 1s 2024-03-29T04:10:09.289 INFO:teuthology.orchestra.run.smithi156.stderr:141600K .......... .......... .......... .......... .......... 84% 34.6M 1s 2024-03-29T04:10:09.290 INFO:teuthology.orchestra.run.smithi156.stderr:141650K .......... .......... .......... .......... .......... 84% 34.2M 1s 2024-03-29T04:10:09.292 INFO:teuthology.orchestra.run.smithi156.stderr:141700K .......... .......... .......... .......... .......... 84% 29.3M 1s 2024-03-29T04:10:09.293 INFO:teuthology.orchestra.run.smithi156.stderr:141750K .......... .......... .......... .......... .......... 84% 33.8M 1s 2024-03-29T04:10:09.295 INFO:teuthology.orchestra.run.smithi156.stderr:141800K .......... .......... .......... .......... .......... 84% 34.0M 1s 2024-03-29T04:10:09.296 INFO:teuthology.orchestra.run.smithi156.stderr:141850K .......... .......... .......... .......... .......... 84% 32.3M 1s 2024-03-29T04:10:09.298 INFO:teuthology.orchestra.run.smithi156.stderr:141900K .......... .......... .......... .......... .......... 84% 28.2M 1s 2024-03-29T04:10:09.300 INFO:teuthology.orchestra.run.smithi156.stderr:141950K .......... .......... .......... .......... .......... 84% 31.8M 1s 2024-03-29T04:10:09.301 INFO:teuthology.orchestra.run.smithi156.stderr:142000K .......... .......... .......... .......... .......... 84% 33.4M 1s 2024-03-29T04:10:09.303 INFO:teuthology.orchestra.run.smithi156.stderr:142050K .......... .......... .......... .......... .......... 84% 32.5M 1s 2024-03-29T04:10:09.304 INFO:teuthology.orchestra.run.smithi156.stderr:142100K .......... .......... .......... .......... .......... 84% 27.7M 1s 2024-03-29T04:10:09.306 INFO:teuthology.orchestra.run.smithi156.stderr:142150K .......... .......... .......... .......... .......... 84% 32.7M 1s 2024-03-29T04:10:09.308 INFO:teuthology.orchestra.run.smithi156.stderr:142200K .......... .......... .......... .......... .......... 84% 32.5M 1s 2024-03-29T04:10:09.309 INFO:teuthology.orchestra.run.smithi156.stderr:142250K .......... .......... .......... .......... .......... 84% 32.5M 1s 2024-03-29T04:10:09.311 INFO:teuthology.orchestra.run.smithi156.stderr:142300K .......... .......... .......... .......... .......... 84% 27.1M 1s 2024-03-29T04:10:09.312 INFO:teuthology.orchestra.run.smithi156.stderr:142350K .......... .......... .......... .......... .......... 84% 32.3M 1s 2024-03-29T04:10:09.314 INFO:teuthology.orchestra.run.smithi156.stderr:142400K .......... .......... .......... .......... .......... 84% 32.7M 1s 2024-03-29T04:10:09.315 INFO:teuthology.orchestra.run.smithi156.stderr:142450K .......... .......... .......... .......... .......... 84% 31.9M 1s 2024-03-29T04:10:09.317 INFO:teuthology.orchestra.run.smithi156.stderr:142500K .......... .......... .......... .......... .......... 84% 27.6M 1s 2024-03-29T04:10:09.318 INFO:teuthology.orchestra.run.smithi156.stderr:142550K .......... .......... .......... .......... .......... 84% 31.2M 1s 2024-03-29T04:10:09.320 INFO:teuthology.orchestra.run.smithi156.stderr:142600K .......... .......... .......... .......... .......... 84% 31.9M 1s 2024-03-29T04:10:09.322 INFO:teuthology.orchestra.run.smithi156.stderr:142650K .......... .......... .......... .......... .......... 84% 31.0M 1s 2024-03-29T04:10:09.323 INFO:teuthology.orchestra.run.smithi156.stderr:142700K .......... .......... .......... .......... .......... 84% 26.3M 1s 2024-03-29T04:10:09.325 INFO:teuthology.orchestra.run.smithi156.stderr:142750K .......... .......... .......... .......... .......... 84% 30.3M 1s 2024-03-29T04:10:09.327 INFO:teuthology.orchestra.run.smithi156.stderr:142800K .......... .......... .......... .......... .......... 84% 30.6M 1s 2024-03-29T04:10:09.328 INFO:teuthology.orchestra.run.smithi156.stderr:142850K .......... .......... .......... .......... .......... 84% 30.6M 1s 2024-03-29T04:10:09.330 INFO:teuthology.orchestra.run.smithi156.stderr:142900K .......... .......... .......... .......... .......... 84% 26.8M 1s 2024-03-29T04:10:09.332 INFO:teuthology.orchestra.run.smithi156.stderr:142950K .......... .......... .......... .......... .......... 84% 31.1M 1s 2024-03-29T04:10:09.333 INFO:teuthology.orchestra.run.smithi156.stderr:143000K .......... .......... .......... .......... .......... 84% 30.8M 1s 2024-03-29T04:10:09.335 INFO:teuthology.orchestra.run.smithi156.stderr:143050K .......... .......... .......... .......... .......... 84% 31.4M 1s 2024-03-29T04:10:09.337 INFO:teuthology.orchestra.run.smithi156.stderr:143100K .......... .......... .......... .......... .......... 84% 26.4M 1s 2024-03-29T04:10:09.338 INFO:teuthology.orchestra.run.smithi156.stderr:143150K .......... .......... .......... .......... .......... 84% 30.8M 1s 2024-03-29T04:10:09.340 INFO:teuthology.orchestra.run.smithi156.stderr:143200K .......... .......... .......... .......... .......... 84% 30.9M 1s 2024-03-29T04:10:09.341 INFO:teuthology.orchestra.run.smithi156.stderr:143250K .......... .......... .......... .......... .......... 85% 30.8M 1s 2024-03-29T04:10:09.343 INFO:teuthology.orchestra.run.smithi156.stderr:143300K .......... .......... .......... .......... .......... 85% 26.3M 1s 2024-03-29T04:10:09.346 INFO:teuthology.orchestra.run.smithi156.stderr:143350K .......... .......... .......... .......... .......... 85% 22.5M 1s 2024-03-29T04:10:09.348 INFO:teuthology.orchestra.run.smithi156.stderr:143400K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-03-29T04:10:09.349 INFO:teuthology.orchestra.run.smithi156.stderr:143450K .......... .......... .......... .......... .......... 85% 27.4M 1s 2024-03-29T04:10:09.351 INFO:teuthology.orchestra.run.smithi156.stderr:143500K .......... .......... .......... .......... .......... 85% 25.5M 1s 2024-03-29T04:10:09.353 INFO:teuthology.orchestra.run.smithi156.stderr:143550K .......... .......... .......... .......... .......... 85% 30.9M 1s 2024-03-29T04:10:09.354 INFO:teuthology.orchestra.run.smithi156.stderr:143600K .......... .......... .......... .......... .......... 85% 30.6M 1s 2024-03-29T04:10:09.356 INFO:teuthology.orchestra.run.smithi156.stderr:143650K .......... .......... .......... .......... .......... 85% 30.1M 1s 2024-03-29T04:10:09.358 INFO:teuthology.orchestra.run.smithi156.stderr:143700K .......... .......... .......... .......... .......... 85% 26.0M 1s 2024-03-29T04:10:09.359 INFO:teuthology.orchestra.run.smithi156.stderr:143750K .......... .......... .......... .......... .......... 85% 32.7M 1s 2024-03-29T04:10:09.361 INFO:teuthology.orchestra.run.smithi156.stderr:143800K .......... .......... .......... .......... .......... 85% 32.5M 1s 2024-03-29T04:10:09.362 INFO:teuthology.orchestra.run.smithi156.stderr:143850K .......... .......... .......... .......... .......... 85% 38.6M 1s 2024-03-29T04:10:09.363 INFO:teuthology.orchestra.run.smithi156.stderr:143900K .......... .......... .......... .......... .......... 85% 32.6M 1s 2024-03-29T04:10:09.365 INFO:teuthology.orchestra.run.smithi156.stderr:143950K .......... .......... .......... .......... .......... 85% 39.4M 1s 2024-03-29T04:10:09.366 INFO:teuthology.orchestra.run.smithi156.stderr:144000K .......... .......... .......... .......... .......... 85% 39.1M 1s 2024-03-29T04:10:09.367 INFO:teuthology.orchestra.run.smithi156.stderr:144050K .......... .......... .......... .......... .......... 85% 38.9M 1s 2024-03-29T04:10:09.369 INFO:teuthology.orchestra.run.smithi156.stderr:144100K .......... .......... .......... .......... .......... 85% 33.5M 1s 2024-03-29T04:10:09.370 INFO:teuthology.orchestra.run.smithi156.stderr:144150K .......... .......... .......... .......... .......... 85% 39.3M 1s 2024-03-29T04:10:09.371 INFO:teuthology.orchestra.run.smithi156.stderr:144200K .......... .......... .......... .......... .......... 85% 38.5M 1s 2024-03-29T04:10:09.373 INFO:teuthology.orchestra.run.smithi156.stderr:144250K .......... .......... .......... .......... .......... 85% 38.8M 1s 2024-03-29T04:10:09.374 INFO:teuthology.orchestra.run.smithi156.stderr:144300K .......... .......... .......... .......... .......... 85% 32.6M 1s 2024-03-29T04:10:09.375 INFO:teuthology.orchestra.run.smithi156.stderr:144350K .......... .......... .......... .......... .......... 85% 37.8M 1s 2024-03-29T04:10:09.377 INFO:teuthology.orchestra.run.smithi156.stderr:144400K .......... .......... .......... .......... .......... 85% 37.8M 1s 2024-03-29T04:10:09.378 INFO:teuthology.orchestra.run.smithi156.stderr:144450K .......... .......... .......... .......... .......... 85% 38.0M 1s 2024-03-29T04:10:09.380 INFO:teuthology.orchestra.run.smithi156.stderr:144500K .......... .......... .......... .......... .......... 85% 31.7M 1s 2024-03-29T04:10:09.381 INFO:teuthology.orchestra.run.smithi156.stderr:144550K .......... .......... .......... .......... .......... 85% 38.7M 1s 2024-03-29T04:10:09.382 INFO:teuthology.orchestra.run.smithi156.stderr:144600K .......... .......... .......... .......... .......... 85% 38.5M 1s 2024-03-29T04:10:09.383 INFO:teuthology.orchestra.run.smithi156.stderr:144650K .......... .......... .......... .......... .......... 85% 37.7M 1s 2024-03-29T04:10:09.385 INFO:teuthology.orchestra.run.smithi156.stderr:144700K .......... .......... .......... .......... .......... 85% 32.3M 1s 2024-03-29T04:10:09.386 INFO:teuthology.orchestra.run.smithi156.stderr:144750K .......... .......... .......... .......... .......... 85% 37.5M 1s 2024-03-29T04:10:09.391 INFO:teuthology.orchestra.run.smithi156.stderr:144800K .......... .......... .......... .......... .......... 85% 37.7M 1s 2024-03-29T04:10:09.391 INFO:teuthology.orchestra.run.smithi156.stderr:144850K .......... .......... .......... .......... .......... 85% 37.1M 1s 2024-03-29T04:10:09.391 INFO:teuthology.orchestra.run.smithi156.stderr:144900K .......... .......... .......... .......... .......... 85% 31.6M 1s 2024-03-29T04:10:09.392 INFO:teuthology.orchestra.run.smithi156.stderr:144950K .......... .......... .......... .......... .......... 86% 36.8M 1s 2024-03-29T04:10:09.393 INFO:teuthology.orchestra.run.smithi156.stderr:145000K .......... .......... .......... .......... .......... 86% 36.7M 1s 2024-03-29T04:10:09.394 INFO:teuthology.orchestra.run.smithi156.stderr:145050K .......... .......... .......... .......... .......... 86% 36.5M 1s 2024-03-29T04:10:09.396 INFO:teuthology.orchestra.run.smithi156.stderr:145100K .......... .......... .......... .......... .......... 86% 31.1M 1s 2024-03-29T04:10:09.397 INFO:teuthology.orchestra.run.smithi156.stderr:145150K .......... .......... .......... .......... .......... 86% 33.9M 1s 2024-03-29T04:10:09.399 INFO:teuthology.orchestra.run.smithi156.stderr:145200K .......... .......... .......... .......... .......... 86% 32.7M 1s 2024-03-29T04:10:09.404 INFO:teuthology.orchestra.run.smithi156.stderr:145250K .......... .......... .......... .......... .......... 86% 32.8M 1s 2024-03-29T04:10:09.404 INFO:teuthology.orchestra.run.smithi156.stderr:145300K .......... .......... .......... .......... .......... 86% 27.8M 1s 2024-03-29T04:10:09.404 INFO:teuthology.orchestra.run.smithi156.stderr:145350K .......... .......... .......... .......... .......... 86% 33.7M 1s 2024-03-29T04:10:09.406 INFO:teuthology.orchestra.run.smithi156.stderr:145400K .......... .......... .......... .......... .......... 86% 33.8M 1s 2024-03-29T04:10:09.407 INFO:teuthology.orchestra.run.smithi156.stderr:145450K .......... .......... .......... .......... .......... 86% 34.6M 1s 2024-03-29T04:10:09.408 INFO:teuthology.orchestra.run.smithi156.stderr:145500K .......... .......... .......... .......... .......... 86% 29.1M 1s 2024-03-29T04:10:09.410 INFO:teuthology.orchestra.run.smithi156.stderr:145550K .......... .......... .......... .......... .......... 86% 32.9M 1s 2024-03-29T04:10:09.411 INFO:teuthology.orchestra.run.smithi156.stderr:145600K .......... .......... .......... .......... .......... 86% 33.5M 1s 2024-03-29T04:10:09.413 INFO:teuthology.orchestra.run.smithi156.stderr:145650K .......... .......... .......... .......... .......... 86% 33.2M 1s 2024-03-29T04:10:09.414 INFO:teuthology.orchestra.run.smithi156.stderr:145700K .......... .......... .......... .......... .......... 86% 28.5M 1s 2024-03-29T04:10:09.416 INFO:teuthology.orchestra.run.smithi156.stderr:145750K .......... .......... .......... .......... .......... 86% 31.3M 1s 2024-03-29T04:10:09.417 INFO:teuthology.orchestra.run.smithi156.stderr:145800K .......... .......... .......... .......... .......... 86% 31.3M 1s 2024-03-29T04:10:09.419 INFO:teuthology.orchestra.run.smithi156.stderr:145850K .......... .......... .......... .......... .......... 86% 31.3M 1s 2024-03-29T04:10:09.421 INFO:teuthology.orchestra.run.smithi156.stderr:145900K .......... .......... .......... .......... .......... 86% 26.6M 1s 2024-03-29T04:10:09.422 INFO:teuthology.orchestra.run.smithi156.stderr:145950K .......... .......... .......... .......... .......... 86% 31.4M 1s 2024-03-29T04:10:09.424 INFO:teuthology.orchestra.run.smithi156.stderr:146000K .......... .......... .......... .......... .......... 86% 31.8M 1s 2024-03-29T04:10:09.426 INFO:teuthology.orchestra.run.smithi156.stderr:146050K .......... .......... .......... .......... .......... 86% 31.8M 1s 2024-03-29T04:10:09.427 INFO:teuthology.orchestra.run.smithi156.stderr:146100K .......... .......... .......... .......... .......... 86% 27.3M 1s 2024-03-29T04:10:09.431 INFO:teuthology.orchestra.run.smithi156.stderr:146150K .......... .......... .......... .......... .......... 86% 31.8M 1s 2024-03-29T04:10:09.431 INFO:teuthology.orchestra.run.smithi156.stderr:146200K .......... .......... .......... .......... .......... 86% 32.0M 1s 2024-03-29T04:10:09.432 INFO:teuthology.orchestra.run.smithi156.stderr:146250K .......... .......... .......... .......... .......... 86% 31.5M 1s 2024-03-29T04:10:09.433 INFO:teuthology.orchestra.run.smithi156.stderr:146300K .......... .......... .......... .......... .......... 86% 26.9M 1s 2024-03-29T04:10:09.435 INFO:teuthology.orchestra.run.smithi156.stderr:146350K .......... .......... .......... .......... .......... 86% 31.4M 1s 2024-03-29T04:10:09.437 INFO:teuthology.orchestra.run.smithi156.stderr:146400K .......... .......... .......... .......... .......... 86% 31.7M 1s 2024-03-29T04:10:09.438 INFO:teuthology.orchestra.run.smithi156.stderr:146450K .......... .......... .......... .......... .......... 86% 31.8M 1s 2024-03-29T04:10:09.440 INFO:teuthology.orchestra.run.smithi156.stderr:146500K .......... .......... .......... .......... .......... 86% 27.0M 1s 2024-03-29T04:10:09.441 INFO:teuthology.orchestra.run.smithi156.stderr:146550K .......... .......... .......... .......... .......... 86% 31.7M 1s 2024-03-29T04:10:09.443 INFO:teuthology.orchestra.run.smithi156.stderr:146600K .......... .......... .......... .......... .......... 86% 29.3M 1s 2024-03-29T04:10:09.445 INFO:teuthology.orchestra.run.smithi156.stderr:146650K .......... .......... .......... .......... .......... 87% 31.4M 1s 2024-03-29T04:10:09.447 INFO:teuthology.orchestra.run.smithi156.stderr:146700K .......... .......... .......... .......... .......... 87% 26.4M 1s 2024-03-29T04:10:09.448 INFO:teuthology.orchestra.run.smithi156.stderr:146750K .......... .......... .......... .......... .......... 87% 29.7M 1s 2024-03-29T04:10:09.450 INFO:teuthology.orchestra.run.smithi156.stderr:146800K .......... .......... .......... .......... .......... 87% 31.0M 1s 2024-03-29T04:10:09.451 INFO:teuthology.orchestra.run.smithi156.stderr:146850K .......... .......... .......... .......... .......... 87% 31.2M 1s 2024-03-29T04:10:09.453 INFO:teuthology.orchestra.run.smithi156.stderr:146900K .......... .......... .......... .......... .......... 87% 26.6M 1s 2024-03-29T04:10:09.455 INFO:teuthology.orchestra.run.smithi156.stderr:146950K .......... .......... .......... .......... .......... 87% 30.0M 1s 2024-03-29T04:10:09.456 INFO:teuthology.orchestra.run.smithi156.stderr:147000K .......... .......... .......... .......... .......... 87% 29.9M 1s 2024-03-29T04:10:09.458 INFO:teuthology.orchestra.run.smithi156.stderr:147050K .......... .......... .......... .......... .......... 87% 29.9M 1s 2024-03-29T04:10:09.460 INFO:teuthology.orchestra.run.smithi156.stderr:147100K .......... .......... .......... .......... .......... 87% 24.7M 1s 2024-03-29T04:10:09.462 INFO:teuthology.orchestra.run.smithi156.stderr:147150K .......... .......... .......... .......... .......... 87% 29.8M 1s 2024-03-29T04:10:09.463 INFO:teuthology.orchestra.run.smithi156.stderr:147200K .......... .......... .......... .......... .......... 87% 29.6M 1s 2024-03-29T04:10:09.465 INFO:teuthology.orchestra.run.smithi156.stderr:147250K .......... .......... .......... .......... .......... 87% 28.7M 1s 2024-03-29T04:10:09.467 INFO:teuthology.orchestra.run.smithi156.stderr:147300K .......... .......... .......... .......... .......... 87% 26.0M 1s 2024-03-29T04:10:09.468 INFO:teuthology.orchestra.run.smithi156.stderr:147350K .......... .......... .......... .......... .......... 87% 29.8M 1s 2024-03-29T04:10:09.470 INFO:teuthology.orchestra.run.smithi156.stderr:147400K .......... .......... .......... .......... .......... 87% 29.3M 1s 2024-03-29T04:10:09.472 INFO:teuthology.orchestra.run.smithi156.stderr:147450K .......... .......... .......... .......... .......... 87% 31.7M 1s 2024-03-29T04:10:09.473 INFO:teuthology.orchestra.run.smithi156.stderr:147500K .......... .......... .......... .......... .......... 87% 30.9M 1s 2024-03-29T04:10:09.475 INFO:teuthology.orchestra.run.smithi156.stderr:147550K .......... .......... .......... .......... .......... 87% 28.7M 1s 2024-03-29T04:10:09.476 INFO:teuthology.orchestra.run.smithi156.stderr:147600K .......... .......... .......... .......... .......... 87% 42.4M 1s 2024-03-29T04:10:09.477 INFO:teuthology.orchestra.run.smithi156.stderr:147650K .......... .......... .......... .......... .......... 87% 42.7M 1s 2024-03-29T04:10:09.478 INFO:teuthology.orchestra.run.smithi156.stderr:147700K .......... .......... .......... .......... .......... 87% 35.9M 1s 2024-03-29T04:10:09.480 INFO:teuthology.orchestra.run.smithi156.stderr:147750K .......... .......... .......... .......... .......... 87% 42.9M 1s 2024-03-29T04:10:09.481 INFO:teuthology.orchestra.run.smithi156.stderr:147800K .......... .......... .......... .......... .......... 87% 42.7M 1s 2024-03-29T04:10:09.482 INFO:teuthology.orchestra.run.smithi156.stderr:147850K .......... .......... .......... .......... .......... 87% 43.0M 1s 2024-03-29T04:10:09.483 INFO:teuthology.orchestra.run.smithi156.stderr:147900K .......... .......... .......... .......... .......... 87% 36.1M 1s 2024-03-29T04:10:09.485 INFO:teuthology.orchestra.run.smithi156.stderr:147950K .......... .......... .......... .......... .......... 87% 42.1M 1s 2024-03-29T04:10:09.486 INFO:teuthology.orchestra.run.smithi156.stderr:148000K .......... .......... .......... .......... .......... 87% 37.7M 1s 2024-03-29T04:10:09.487 INFO:teuthology.orchestra.run.smithi156.stderr:148050K .......... .......... .......... .......... .......... 87% 39.9M 1s 2024-03-29T04:10:09.488 INFO:teuthology.orchestra.run.smithi156.stderr:148100K .......... .......... .......... .......... .......... 87% 35.4M 1s 2024-03-29T04:10:09.490 INFO:teuthology.orchestra.run.smithi156.stderr:148150K .......... .......... .......... .......... .......... 87% 34.4M 1s 2024-03-29T04:10:09.491 INFO:teuthology.orchestra.run.smithi156.stderr:148200K .......... .......... .......... .......... .......... 87% 40.2M 1s 2024-03-29T04:10:09.492 INFO:teuthology.orchestra.run.smithi156.stderr:148250K .......... .......... .......... .......... .......... 87% 39.4M 1s 2024-03-29T04:10:09.494 INFO:teuthology.orchestra.run.smithi156.stderr:148300K .......... .......... .......... .......... .......... 88% 33.8M 1s 2024-03-29T04:10:09.495 INFO:teuthology.orchestra.run.smithi156.stderr:148350K .......... .......... .......... .......... .......... 88% 39.6M 1s 2024-03-29T04:10:09.496 INFO:teuthology.orchestra.run.smithi156.stderr:148400K .......... .......... .......... .......... .......... 88% 39.6M 1s 2024-03-29T04:10:09.498 INFO:teuthology.orchestra.run.smithi156.stderr:148450K .......... .......... .......... .......... .......... 88% 36.7M 1s 2024-03-29T04:10:09.499 INFO:teuthology.orchestra.run.smithi156.stderr:148500K .......... .......... .......... .......... .......... 88% 30.0M 1s 2024-03-29T04:10:09.501 INFO:teuthology.orchestra.run.smithi156.stderr:148550K .......... .......... .......... .......... .......... 88% 36.6M 1s 2024-03-29T04:10:09.502 INFO:teuthology.orchestra.run.smithi156.stderr:148600K .......... .......... .......... .......... .......... 88% 36.3M 1s 2024-03-29T04:10:09.503 INFO:teuthology.orchestra.run.smithi156.stderr:148650K .......... .......... .......... .......... .......... 88% 37.0M 1s 2024-03-29T04:10:09.505 INFO:teuthology.orchestra.run.smithi156.stderr:148700K .......... .......... .......... .......... .......... 88% 31.0M 1s 2024-03-29T04:10:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:148750K .......... .......... .......... .......... .......... 88% 36.5M 1s 2024-03-29T04:10:09.507 INFO:teuthology.orchestra.run.smithi156.stderr:148800K .......... .......... .......... .......... .......... 88% 35.7M 1s 2024-03-29T04:10:09.509 INFO:teuthology.orchestra.run.smithi156.stderr:148850K .......... .......... .......... .......... .......... 88% 35.9M 1s 2024-03-29T04:10:09.510 INFO:teuthology.orchestra.run.smithi156.stderr:148900K .......... .......... .......... .......... .......... 88% 31.3M 1s 2024-03-29T04:10:09.512 INFO:teuthology.orchestra.run.smithi156.stderr:148950K .......... .......... .......... .......... .......... 88% 36.2M 1s 2024-03-29T04:10:09.513 INFO:teuthology.orchestra.run.smithi156.stderr:149000K .......... .......... .......... .......... .......... 88% 33.8M 1s 2024-03-29T04:10:09.515 INFO:teuthology.orchestra.run.smithi156.stderr:149050K .......... .......... .......... .......... .......... 88% 34.3M 1s 2024-03-29T04:10:09.516 INFO:teuthology.orchestra.run.smithi156.stderr:149100K .......... .......... .......... .......... .......... 88% 29.6M 1s 2024-03-29T04:10:09.518 INFO:teuthology.orchestra.run.smithi156.stderr:149150K .......... .......... .......... .......... .......... 88% 34.0M 1s 2024-03-29T04:10:09.519 INFO:teuthology.orchestra.run.smithi156.stderr:149200K .......... .......... .......... .......... .......... 88% 34.2M 1s 2024-03-29T04:10:09.521 INFO:teuthology.orchestra.run.smithi156.stderr:149250K .......... .......... .......... .......... .......... 88% 33.9M 1s 2024-03-29T04:10:09.522 INFO:teuthology.orchestra.run.smithi156.stderr:149300K .......... .......... .......... .......... .......... 88% 29.5M 1s 2024-03-29T04:10:09.524 INFO:teuthology.orchestra.run.smithi156.stderr:149350K .......... .......... .......... .......... .......... 88% 35.2M 1s 2024-03-29T04:10:09.525 INFO:teuthology.orchestra.run.smithi156.stderr:149400K .......... .......... .......... .......... .......... 88% 33.0M 1s 2024-03-29T04:10:09.527 INFO:teuthology.orchestra.run.smithi156.stderr:149450K .......... .......... .......... .......... .......... 88% 34.9M 1s 2024-03-29T04:10:09.528 INFO:teuthology.orchestra.run.smithi156.stderr:149500K .......... .......... .......... .......... .......... 88% 27.6M 1s 2024-03-29T04:10:09.530 INFO:teuthology.orchestra.run.smithi156.stderr:149550K .......... .......... .......... .......... .......... 88% 28.3M 1s 2024-03-29T04:10:09.532 INFO:teuthology.orchestra.run.smithi156.stderr:149600K .......... .......... .......... .......... .......... 88% 29.0M 1s 2024-03-29T04:10:09.533 INFO:teuthology.orchestra.run.smithi156.stderr:149650K .......... .......... .......... .......... .......... 88% 28.6M 1s 2024-03-29T04:10:09.535 INFO:teuthology.orchestra.run.smithi156.stderr:149700K .......... .......... .......... .......... .......... 88% 25.1M 1s 2024-03-29T04:10:09.537 INFO:teuthology.orchestra.run.smithi156.stderr:149750K .......... .......... .......... .......... .......... 88% 34.6M 1s 2024-03-29T04:10:09.538 INFO:teuthology.orchestra.run.smithi156.stderr:149800K .......... .......... .......... .......... .......... 88% 34.9M 1s 2024-03-29T04:10:09.540 INFO:teuthology.orchestra.run.smithi156.stderr:149850K .......... .......... .......... .......... .......... 88% 35.1M 1s 2024-03-29T04:10:09.541 INFO:teuthology.orchestra.run.smithi156.stderr:149900K .......... .......... .......... .......... .......... 88% 29.3M 1s 2024-03-29T04:10:09.542 INFO:teuthology.orchestra.run.smithi156.stderr:149950K .......... .......... .......... .......... .......... 88% 33.3M 1s 2024-03-29T04:10:09.544 INFO:teuthology.orchestra.run.smithi156.stderr:150000K .......... .......... .......... .......... .......... 89% 31.2M 1s 2024-03-29T04:10:09.546 INFO:teuthology.orchestra.run.smithi156.stderr:150050K .......... .......... .......... .......... .......... 89% 33.1M 1s 2024-03-29T04:10:09.547 INFO:teuthology.orchestra.run.smithi156.stderr:150100K .......... .......... .......... .......... .......... 89% 27.1M 1s 2024-03-29T04:10:09.549 INFO:teuthology.orchestra.run.smithi156.stderr:150150K .......... .......... .......... .......... .......... 89% 29.8M 1s 2024-03-29T04:10:09.551 INFO:teuthology.orchestra.run.smithi156.stderr:150200K .......... .......... .......... .......... .......... 89% 31.4M 1s 2024-03-29T04:10:09.552 INFO:teuthology.orchestra.run.smithi156.stderr:150250K .......... .......... .......... .......... .......... 89% 30.1M 1s 2024-03-29T04:10:09.554 INFO:teuthology.orchestra.run.smithi156.stderr:150300K .......... .......... .......... .......... .......... 89% 26.3M 1s 2024-03-29T04:10:09.556 INFO:teuthology.orchestra.run.smithi156.stderr:150350K .......... .......... .......... .......... .......... 89% 30.8M 1s 2024-03-29T04:10:09.557 INFO:teuthology.orchestra.run.smithi156.stderr:150400K .......... .......... .......... .......... .......... 89% 30.3M 1s 2024-03-29T04:10:09.559 INFO:teuthology.orchestra.run.smithi156.stderr:150450K .......... .......... .......... .......... .......... 89% 30.9M 1s 2024-03-29T04:10:09.561 INFO:teuthology.orchestra.run.smithi156.stderr:150500K .......... .......... .......... .......... .......... 89% 25.6M 1s 2024-03-29T04:10:09.562 INFO:teuthology.orchestra.run.smithi156.stderr:150550K .......... .......... .......... .......... .......... 89% 30.2M 1s 2024-03-29T04:10:09.564 INFO:teuthology.orchestra.run.smithi156.stderr:150600K .......... .......... .......... .......... .......... 89% 30.2M 1s 2024-03-29T04:10:09.566 INFO:teuthology.orchestra.run.smithi156.stderr:150650K .......... .......... .......... .......... .......... 89% 30.3M 1s 2024-03-29T04:10:09.568 INFO:teuthology.orchestra.run.smithi156.stderr:150700K .......... .......... .......... .......... .......... 89% 25.8M 1s 2024-03-29T04:10:09.569 INFO:teuthology.orchestra.run.smithi156.stderr:150750K .......... .......... .......... .......... .......... 89% 29.8M 1s 2024-03-29T04:10:09.571 INFO:teuthology.orchestra.run.smithi156.stderr:150800K .......... .......... .......... .......... .......... 89% 29.2M 1s 2024-03-29T04:10:09.573 INFO:teuthology.orchestra.run.smithi156.stderr:150850K .......... .......... .......... .......... .......... 89% 29.8M 1s 2024-03-29T04:10:09.574 INFO:teuthology.orchestra.run.smithi156.stderr:150900K .......... .......... .......... .......... .......... 89% 24.3M 1s 2024-03-29T04:10:09.576 INFO:teuthology.orchestra.run.smithi156.stderr:150950K .......... .......... .......... .......... .......... 89% 29.0M 1s 2024-03-29T04:10:09.578 INFO:teuthology.orchestra.run.smithi156.stderr:151000K .......... .......... .......... .......... .......... 89% 29.3M 1s 2024-03-29T04:10:09.580 INFO:teuthology.orchestra.run.smithi156.stderr:151050K .......... .......... .......... .......... .......... 89% 27.7M 1s 2024-03-29T04:10:09.582 INFO:teuthology.orchestra.run.smithi156.stderr:151100K .......... .......... .......... .......... .......... 89% 24.8M 1s 2024-03-29T04:10:09.583 INFO:teuthology.orchestra.run.smithi156.stderr:151150K .......... .......... .......... .......... .......... 89% 28.6M 1s 2024-03-29T04:10:09.585 INFO:teuthology.orchestra.run.smithi156.stderr:151200K .......... .......... .......... .......... .......... 89% 28.2M 1s 2024-03-29T04:10:09.587 INFO:teuthology.orchestra.run.smithi156.stderr:151250K .......... .......... .......... .......... .......... 89% 28.3M 1s 2024-03-29T04:10:09.589 INFO:teuthology.orchestra.run.smithi156.stderr:151300K .......... .......... .......... .......... .......... 89% 24.6M 1s 2024-03-29T04:10:09.590 INFO:teuthology.orchestra.run.smithi156.stderr:151350K .......... .......... .......... .......... .......... 89% 28.2M 1s 2024-03-29T04:10:09.592 INFO:teuthology.orchestra.run.smithi156.stderr:151400K .......... .......... .......... .......... .......... 89% 28.9M 1s 2024-03-29T04:10:09.594 INFO:teuthology.orchestra.run.smithi156.stderr:151450K .......... .......... .......... .......... .......... 89% 27.6M 1s 2024-03-29T04:10:09.596 INFO:teuthology.orchestra.run.smithi156.stderr:151500K .......... .......... .......... .......... .......... 89% 21.0M 1s 2024-03-29T04:10:09.598 INFO:teuthology.orchestra.run.smithi156.stderr:151550K .......... .......... .......... .......... .......... 89% 30.6M 1s 2024-03-29T04:10:09.599 INFO:teuthology.orchestra.run.smithi156.stderr:151600K .......... .......... .......... .......... .......... 89% 37.3M 1s 2024-03-29T04:10:09.601 INFO:teuthology.orchestra.run.smithi156.stderr:151650K .......... .......... .......... .......... .......... 89% 35.2M 1s 2024-03-29T04:10:09.603 INFO:teuthology.orchestra.run.smithi156.stderr:151700K .......... .......... .......... .......... .......... 90% 24.1M 1s 2024-03-29T04:10:09.604 INFO:teuthology.orchestra.run.smithi156.stderr:151750K .......... .......... .......... .......... .......... 90% 29.6M 1s 2024-03-29T04:10:09.606 INFO:teuthology.orchestra.run.smithi156.stderr:151800K .......... .......... .......... .......... .......... 90% 31.3M 1s 2024-03-29T04:10:09.607 INFO:teuthology.orchestra.run.smithi156.stderr:151850K .......... .......... .......... .......... .......... 90% 32.2M 1s 2024-03-29T04:10:09.609 INFO:teuthology.orchestra.run.smithi156.stderr:151900K .......... .......... .......... .......... .......... 90% 28.9M 1s 2024-03-29T04:10:09.611 INFO:teuthology.orchestra.run.smithi156.stderr:151950K .......... .......... .......... .......... .......... 90% 29.2M 1s 2024-03-29T04:10:09.612 INFO:teuthology.orchestra.run.smithi156.stderr:152000K .......... .......... .......... .......... .......... 90% 29.0M 1s 2024-03-29T04:10:09.614 INFO:teuthology.orchestra.run.smithi156.stderr:152050K .......... .......... .......... .......... .......... 90% 27.4M 1s 2024-03-29T04:10:09.616 INFO:teuthology.orchestra.run.smithi156.stderr:152100K .......... .......... .......... .......... .......... 90% 24.6M 1s 2024-03-29T04:10:09.618 INFO:teuthology.orchestra.run.smithi156.stderr:152150K .......... .......... .......... .......... .......... 90% 28.7M 1s 2024-03-29T04:10:09.620 INFO:teuthology.orchestra.run.smithi156.stderr:152200K .......... .......... .......... .......... .......... 90% 27.1M 1s 2024-03-29T04:10:09.622 INFO:teuthology.orchestra.run.smithi156.stderr:152250K .......... .......... .......... .......... .......... 90% 26.7M 1s 2024-03-29T04:10:09.623 INFO:teuthology.orchestra.run.smithi156.stderr:152300K .......... .......... .......... .......... .......... 90% 25.8M 1s 2024-03-29T04:10:09.625 INFO:teuthology.orchestra.run.smithi156.stderr:152350K .......... .......... .......... .......... .......... 90% 26.2M 1s 2024-03-29T04:10:09.627 INFO:teuthology.orchestra.run.smithi156.stderr:152400K .......... .......... .......... .......... .......... 90% 26.9M 1s 2024-03-29T04:10:09.629 INFO:teuthology.orchestra.run.smithi156.stderr:152450K .......... .......... .......... .......... .......... 90% 26.1M 1s 2024-03-29T04:10:09.631 INFO:teuthology.orchestra.run.smithi156.stderr:152500K .......... .......... .......... .......... .......... 90% 23.5M 1s 2024-03-29T04:10:09.633 INFO:teuthology.orchestra.run.smithi156.stderr:152550K .......... .......... .......... .......... .......... 90% 28.8M 1s 2024-03-29T04:10:09.634 INFO:teuthology.orchestra.run.smithi156.stderr:152600K .......... .......... .......... .......... .......... 90% 29.8M 1s 2024-03-29T04:10:09.636 INFO:teuthology.orchestra.run.smithi156.stderr:152650K .......... .......... .......... .......... .......... 90% 29.0M 1s 2024-03-29T04:10:09.638 INFO:teuthology.orchestra.run.smithi156.stderr:152700K .......... .......... .......... .......... .......... 90% 25.5M 1s 2024-03-29T04:10:09.640 INFO:teuthology.orchestra.run.smithi156.stderr:152750K .......... .......... .......... .......... .......... 90% 25.5M 1s 2024-03-29T04:10:09.642 INFO:teuthology.orchestra.run.smithi156.stderr:152800K .......... .......... .......... .......... .......... 90% 25.7M 1s 2024-03-29T04:10:09.644 INFO:teuthology.orchestra.run.smithi156.stderr:152850K .......... .......... .......... .......... .......... 90% 25.2M 1s 2024-03-29T04:10:09.646 INFO:teuthology.orchestra.run.smithi156.stderr:152900K .......... .......... .......... .......... .......... 90% 22.0M 1s 2024-03-29T04:10:09.648 INFO:teuthology.orchestra.run.smithi156.stderr:152950K .......... .......... .......... .......... .......... 90% 29.2M 1s 2024-03-29T04:10:09.649 INFO:teuthology.orchestra.run.smithi156.stderr:153000K .......... .......... .......... .......... .......... 90% 30.2M 1s 2024-03-29T04:10:09.651 INFO:teuthology.orchestra.run.smithi156.stderr:153050K .......... .......... .......... .......... .......... 90% 30.4M 1s 2024-03-29T04:10:09.653 INFO:teuthology.orchestra.run.smithi156.stderr:153100K .......... .......... .......... .......... .......... 90% 26.9M 1s 2024-03-29T04:10:09.654 INFO:teuthology.orchestra.run.smithi156.stderr:153150K .......... .......... .......... .......... .......... 90% 30.6M 1s 2024-03-29T04:10:09.655 INFO:teuthology.orchestra.run.smithi156.stderr:153200K .......... .......... .......... .......... .......... 90% 35.3M 1s 2024-03-29T04:10:09.657 INFO:teuthology.orchestra.run.smithi156.stderr:153250K .......... .......... .......... .......... .......... 90% 34.0M 1s 2024-03-29T04:10:09.659 INFO:teuthology.orchestra.run.smithi156.stderr:153300K .......... .......... .......... .......... .......... 90% 29.9M 1s 2024-03-29T04:10:09.660 INFO:teuthology.orchestra.run.smithi156.stderr:153350K .......... .......... .......... .......... .......... 91% 34.6M 1s 2024-03-29T04:10:09.661 INFO:teuthology.orchestra.run.smithi156.stderr:153400K .......... .......... .......... .......... .......... 91% 34.8M 1s 2024-03-29T04:10:09.663 INFO:teuthology.orchestra.run.smithi156.stderr:153450K .......... .......... .......... .......... .......... 91% 34.5M 1s 2024-03-29T04:10:09.665 INFO:teuthology.orchestra.run.smithi156.stderr:153500K .......... .......... .......... .......... .......... 91% 29.7M 1s 2024-03-29T04:10:09.666 INFO:teuthology.orchestra.run.smithi156.stderr:153550K .......... .......... .......... .......... .......... 91% 33.9M 1s 2024-03-29T04:10:09.667 INFO:teuthology.orchestra.run.smithi156.stderr:153600K .......... .......... .......... .......... .......... 91% 33.8M 1s 2024-03-29T04:10:09.669 INFO:teuthology.orchestra.run.smithi156.stderr:153650K .......... .......... .......... .......... .......... 91% 33.8M 1s 2024-03-29T04:10:09.671 INFO:teuthology.orchestra.run.smithi156.stderr:153700K .......... .......... .......... .......... .......... 91% 28.8M 1s 2024-03-29T04:10:09.672 INFO:teuthology.orchestra.run.smithi156.stderr:153750K .......... .......... .......... .......... .......... 91% 33.6M 1s 2024-03-29T04:10:09.674 INFO:teuthology.orchestra.run.smithi156.stderr:153800K .......... .......... .......... .......... .......... 91% 32.6M 1s 2024-03-29T04:10:09.675 INFO:teuthology.orchestra.run.smithi156.stderr:153850K .......... .......... .......... .......... .......... 91% 33.1M 1s 2024-03-29T04:10:09.677 INFO:teuthology.orchestra.run.smithi156.stderr:153900K .......... .......... .......... .......... .......... 91% 28.6M 1s 2024-03-29T04:10:09.678 INFO:teuthology.orchestra.run.smithi156.stderr:153950K .......... .......... .......... .......... .......... 91% 32.8M 1s 2024-03-29T04:10:09.680 INFO:teuthology.orchestra.run.smithi156.stderr:154000K .......... .......... .......... .......... .......... 91% 32.0M 1s 2024-03-29T04:10:09.682 INFO:teuthology.orchestra.run.smithi156.stderr:154050K .......... .......... .......... .......... .......... 91% 31.7M 1s 2024-03-29T04:10:09.683 INFO:teuthology.orchestra.run.smithi156.stderr:154100K .......... .......... .......... .......... .......... 91% 27.0M 1s 2024-03-29T04:10:09.685 INFO:teuthology.orchestra.run.smithi156.stderr:154150K .......... .......... .......... .......... .......... 91% 32.1M 1s 2024-03-29T04:10:09.686 INFO:teuthology.orchestra.run.smithi156.stderr:154200K .......... .......... .......... .......... .......... 91% 31.4M 1s 2024-03-29T04:10:09.688 INFO:teuthology.orchestra.run.smithi156.stderr:154250K .......... .......... .......... .......... .......... 91% 31.6M 1s 2024-03-29T04:10:09.690 INFO:teuthology.orchestra.run.smithi156.stderr:154300K .......... .......... .......... .......... .......... 91% 26.6M 1s 2024-03-29T04:10:09.691 INFO:teuthology.orchestra.run.smithi156.stderr:154350K .......... .......... .......... .......... .......... 91% 31.3M 1s 2024-03-29T04:10:09.693 INFO:teuthology.orchestra.run.smithi156.stderr:154400K .......... .......... .......... .......... .......... 91% 31.7M 1s 2024-03-29T04:10:09.694 INFO:teuthology.orchestra.run.smithi156.stderr:154450K .......... .......... .......... .......... .......... 91% 31.7M 1s 2024-03-29T04:10:09.696 INFO:teuthology.orchestra.run.smithi156.stderr:154500K .......... .......... .......... .......... .......... 91% 27.5M 0s 2024-03-29T04:10:09.697 INFO:teuthology.orchestra.run.smithi156.stderr:154550K .......... .......... .......... .......... .......... 91% 32.1M 0s 2024-03-29T04:10:09.699 INFO:teuthology.orchestra.run.smithi156.stderr:154600K .......... .......... .......... .......... .......... 91% 31.6M 0s 2024-03-29T04:10:09.701 INFO:teuthology.orchestra.run.smithi156.stderr:154650K .......... .......... .......... .......... .......... 91% 31.7M 0s 2024-03-29T04:10:09.702 INFO:teuthology.orchestra.run.smithi156.stderr:154700K .......... .......... .......... .......... .......... 91% 27.0M 0s 2024-03-29T04:10:09.704 INFO:teuthology.orchestra.run.smithi156.stderr:154750K .......... .......... .......... .......... .......... 91% 31.2M 0s 2024-03-29T04:10:09.705 INFO:teuthology.orchestra.run.smithi156.stderr:154800K .......... .......... .......... .......... .......... 91% 31.0M 0s 2024-03-29T04:10:09.707 INFO:teuthology.orchestra.run.smithi156.stderr:154850K .......... .......... .......... .......... .......... 91% 30.8M 0s 2024-03-29T04:10:09.709 INFO:teuthology.orchestra.run.smithi156.stderr:154900K .......... .......... .......... .......... .......... 91% 26.5M 0s 2024-03-29T04:10:09.710 INFO:teuthology.orchestra.run.smithi156.stderr:154950K .......... .......... .......... .......... .......... 91% 30.9M 0s 2024-03-29T04:10:09.712 INFO:teuthology.orchestra.run.smithi156.stderr:155000K .......... .......... .......... .......... .......... 91% 30.5M 0s 2024-03-29T04:10:09.714 INFO:teuthology.orchestra.run.smithi156.stderr:155050K .......... .......... .......... .......... .......... 92% 30.1M 0s 2024-03-29T04:10:09.716 INFO:teuthology.orchestra.run.smithi156.stderr:155100K .......... .......... .......... .......... .......... 92% 25.8M 0s 2024-03-29T04:10:09.717 INFO:teuthology.orchestra.run.smithi156.stderr:155150K .......... .......... .......... .......... .......... 92% 29.9M 0s 2024-03-29T04:10:09.719 INFO:teuthology.orchestra.run.smithi156.stderr:155200K .......... .......... .......... .......... .......... 92% 30.2M 0s 2024-03-29T04:10:09.721 INFO:teuthology.orchestra.run.smithi156.stderr:155250K .......... .......... .......... .......... .......... 92% 29.7M 0s 2024-03-29T04:10:09.722 INFO:teuthology.orchestra.run.smithi156.stderr:155300K .......... .......... .......... .......... .......... 92% 25.6M 0s 2024-03-29T04:10:09.724 INFO:teuthology.orchestra.run.smithi156.stderr:155350K .......... .......... .......... .......... .......... 92% 29.3M 0s 2024-03-29T04:10:09.726 INFO:teuthology.orchestra.run.smithi156.stderr:155400K .......... .......... .......... .......... .......... 92% 28.8M 0s 2024-03-29T04:10:09.728 INFO:teuthology.orchestra.run.smithi156.stderr:155450K .......... .......... .......... .......... .......... 92% 26.4M 0s 2024-03-29T04:10:09.730 INFO:teuthology.orchestra.run.smithi156.stderr:155500K .......... .......... .......... .......... .......... 92% 24.1M 0s 2024-03-29T04:10:09.731 INFO:teuthology.orchestra.run.smithi156.stderr:155550K .......... .......... .......... .......... .......... 92% 28.7M 0s 2024-03-29T04:10:09.733 INFO:teuthology.orchestra.run.smithi156.stderr:155600K .......... .......... .......... .......... .......... 92% 28.1M 0s 2024-03-29T04:10:09.736 INFO:teuthology.orchestra.run.smithi156.stderr:155650K .......... .......... .......... .......... .......... 92% 26.2M 0s 2024-03-29T04:10:09.737 INFO:teuthology.orchestra.run.smithi156.stderr:155700K .......... .......... .......... .......... .......... 92% 28.8M 0s 2024-03-29T04:10:09.738 INFO:teuthology.orchestra.run.smithi156.stderr:155750K .......... .......... .......... .......... .......... 92% 29.0M 0s 2024-03-29T04:10:09.740 INFO:teuthology.orchestra.run.smithi156.stderr:155800K .......... .......... .......... .......... .......... 92% 34.9M 0s 2024-03-29T04:10:09.741 INFO:teuthology.orchestra.run.smithi156.stderr:155850K .......... .......... .......... .......... .......... 92% 34.7M 0s 2024-03-29T04:10:09.743 INFO:teuthology.orchestra.run.smithi156.stderr:155900K .......... .......... .......... .......... .......... 92% 28.8M 0s 2024-03-29T04:10:09.744 INFO:teuthology.orchestra.run.smithi156.stderr:155950K .......... .......... .......... .......... .......... 92% 35.7M 0s 2024-03-29T04:10:09.746 INFO:teuthology.orchestra.run.smithi156.stderr:156000K .......... .......... .......... .......... .......... 92% 25.8M 0s 2024-03-29T04:10:09.748 INFO:teuthology.orchestra.run.smithi156.stderr:156050K .......... .......... .......... .......... .......... 92% 26.7M 0s 2024-03-29T04:10:09.750 INFO:teuthology.orchestra.run.smithi156.stderr:156100K .......... .......... .......... .......... .......... 92% 27.1M 0s 2024-03-29T04:10:09.751 INFO:teuthology.orchestra.run.smithi156.stderr:156150K .......... .......... .......... .......... .......... 92% 32.3M 0s 2024-03-29T04:10:09.752 INFO:teuthology.orchestra.run.smithi156.stderr:156200K .......... .......... .......... .......... .......... 92% 34.7M 0s 2024-03-29T04:10:09.755 INFO:teuthology.orchestra.run.smithi156.stderr:156250K .......... .......... .......... .......... .......... 92% 37.5M 0s 2024-03-29T04:10:09.756 INFO:teuthology.orchestra.run.smithi156.stderr:156300K .......... .......... .......... .......... .......... 92% 28.3M 0s 2024-03-29T04:10:09.757 INFO:teuthology.orchestra.run.smithi156.stderr:156350K .......... .......... .......... .......... .......... 92% 33.3M 0s 2024-03-29T04:10:09.759 INFO:teuthology.orchestra.run.smithi156.stderr:156400K .......... .......... .......... .......... .......... 92% 33.1M 0s 2024-03-29T04:10:09.760 INFO:teuthology.orchestra.run.smithi156.stderr:156450K .......... .......... .......... .......... .......... 92% 32.7M 0s 2024-03-29T04:10:09.762 INFO:teuthology.orchestra.run.smithi156.stderr:156500K .......... .......... .......... .......... .......... 92% 28.2M 0s 2024-03-29T04:10:09.763 INFO:teuthology.orchestra.run.smithi156.stderr:156550K .......... .......... .......... .......... .......... 92% 33.4M 0s 2024-03-29T04:10:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:156600K .......... .......... .......... .......... .......... 92% 33.7M 0s 2024-03-29T04:10:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:156650K .......... .......... .......... .......... .......... 92% 33.4M 0s 2024-03-29T04:10:09.768 INFO:teuthology.orchestra.run.smithi156.stderr:156700K .......... .......... .......... .......... .......... 92% 28.7M 0s 2024-03-29T04:10:09.769 INFO:teuthology.orchestra.run.smithi156.stderr:156750K .......... .......... .......... .......... .......... 93% 33.3M 0s 2024-03-29T04:10:09.771 INFO:teuthology.orchestra.run.smithi156.stderr:156800K .......... .......... .......... .......... .......... 93% 33.2M 0s 2024-03-29T04:10:09.773 INFO:teuthology.orchestra.run.smithi156.stderr:156850K .......... .......... .......... .......... .......... 93% 32.8M 0s 2024-03-29T04:10:09.775 INFO:teuthology.orchestra.run.smithi156.stderr:156900K .......... .......... .......... .......... .......... 93% 27.7M 0s 2024-03-29T04:10:09.776 INFO:teuthology.orchestra.run.smithi156.stderr:156950K .......... .......... .......... .......... .......... 93% 33.0M 0s 2024-03-29T04:10:09.777 INFO:teuthology.orchestra.run.smithi156.stderr:157000K .......... .......... .......... .......... .......... 93% 31.9M 0s 2024-03-29T04:10:09.779 INFO:teuthology.orchestra.run.smithi156.stderr:157050K .......... .......... .......... .......... .......... 93% 33.1M 0s 2024-03-29T04:10:09.781 INFO:teuthology.orchestra.run.smithi156.stderr:157100K .......... .......... .......... .......... .......... 93% 27.7M 0s 2024-03-29T04:10:09.783 INFO:teuthology.orchestra.run.smithi156.stderr:157150K .......... .......... .......... .......... .......... 93% 31.9M 0s 2024-03-29T04:10:09.784 INFO:teuthology.orchestra.run.smithi156.stderr:157200K .......... .......... .......... .......... .......... 93% 32.3M 0s 2024-03-29T04:10:09.785 INFO:teuthology.orchestra.run.smithi156.stderr:157250K .......... .......... .......... .......... .......... 93% 32.2M 0s 2024-03-29T04:10:09.787 INFO:teuthology.orchestra.run.smithi156.stderr:157300K .......... .......... .......... .......... .......... 93% 27.1M 0s 2024-03-29T04:10:09.788 INFO:teuthology.orchestra.run.smithi156.stderr:157350K .......... .......... .......... .......... .......... 93% 30.6M 0s 2024-03-29T04:10:09.790 INFO:teuthology.orchestra.run.smithi156.stderr:157400K .......... .......... .......... .......... .......... 93% 31.0M 0s 2024-03-29T04:10:09.792 INFO:teuthology.orchestra.run.smithi156.stderr:157450K .......... .......... .......... .......... .......... 93% 30.6M 0s 2024-03-29T04:10:09.793 INFO:teuthology.orchestra.run.smithi156.stderr:157500K .......... .......... .......... .......... .......... 93% 26.4M 0s 2024-03-29T04:10:09.795 INFO:teuthology.orchestra.run.smithi156.stderr:157550K .......... .......... .......... .......... .......... 93% 30.8M 0s 2024-03-29T04:10:09.801 INFO:teuthology.orchestra.run.smithi156.stderr:157600K .......... .......... .......... .......... .......... 93% 30.7M 0s 2024-03-29T04:10:09.801 INFO:teuthology.orchestra.run.smithi156.stderr:157650K .......... .......... .......... .......... .......... 93% 31.0M 0s 2024-03-29T04:10:09.801 INFO:teuthology.orchestra.run.smithi156.stderr:157700K .......... .......... .......... .......... .......... 93% 26.4M 0s 2024-03-29T04:10:09.802 INFO:teuthology.orchestra.run.smithi156.stderr:157750K .......... .......... .......... .......... .......... 93% 30.8M 0s 2024-03-29T04:10:09.803 INFO:teuthology.orchestra.run.smithi156.stderr:157800K .......... .......... .......... .......... .......... 93% 30.3M 0s 2024-03-29T04:10:09.805 INFO:teuthology.orchestra.run.smithi156.stderr:157850K .......... .......... .......... .......... .......... 93% 31.0M 0s 2024-03-29T04:10:09.806 INFO:teuthology.orchestra.run.smithi156.stderr:157900K .......... .......... .......... .......... .......... 93% 27.7M 0s 2024-03-29T04:10:09.808 INFO:teuthology.orchestra.run.smithi156.stderr:157950K .......... .......... .......... .......... .......... 93% 27.9M 0s 2024-03-29T04:10:09.810 INFO:teuthology.orchestra.run.smithi156.stderr:158000K .......... .......... .......... .......... .......... 93% 34.3M 0s 2024-03-29T04:10:09.811 INFO:teuthology.orchestra.run.smithi156.stderr:158050K .......... .......... .......... .......... .......... 93% 33.6M 0s 2024-03-29T04:10:09.813 INFO:teuthology.orchestra.run.smithi156.stderr:158100K .......... .......... .......... .......... .......... 93% 29.0M 0s 2024-03-29T04:10:09.814 INFO:teuthology.orchestra.run.smithi156.stderr:158150K .......... .......... .......... .......... .......... 93% 33.4M 0s 2024-03-29T04:10:09.816 INFO:teuthology.orchestra.run.smithi156.stderr:158200K .......... .......... .......... .......... .......... 93% 34.1M 0s 2024-03-29T04:10:09.817 INFO:teuthology.orchestra.run.smithi156.stderr:158250K .......... .......... .......... .......... .......... 93% 33.3M 0s 2024-03-29T04:10:09.819 INFO:teuthology.orchestra.run.smithi156.stderr:158300K .......... .......... .......... .......... .......... 93% 29.0M 0s 2024-03-29T04:10:09.820 INFO:teuthology.orchestra.run.smithi156.stderr:158350K .......... .......... .......... .......... .......... 93% 33.3M 0s 2024-03-29T04:10:09.822 INFO:teuthology.orchestra.run.smithi156.stderr:158400K .......... .......... .......... .......... .......... 93% 33.1M 0s 2024-03-29T04:10:09.823 INFO:teuthology.orchestra.run.smithi156.stderr:158450K .......... .......... .......... .......... .......... 94% 33.0M 0s 2024-03-29T04:10:09.825 INFO:teuthology.orchestra.run.smithi156.stderr:158500K .......... .......... .......... .......... .......... 94% 26.2M 0s 2024-03-29T04:10:09.827 INFO:teuthology.orchestra.run.smithi156.stderr:158550K .......... .......... .......... .......... .......... 94% 31.6M 0s 2024-03-29T04:10:09.828 INFO:teuthology.orchestra.run.smithi156.stderr:158600K .......... .......... .......... .......... .......... 94% 32.3M 0s 2024-03-29T04:10:09.830 INFO:teuthology.orchestra.run.smithi156.stderr:158650K .......... .......... .......... .......... .......... 94% 26.4M 0s 2024-03-29T04:10:09.832 INFO:teuthology.orchestra.run.smithi156.stderr:158700K .......... .......... .......... .......... .......... 94% 21.5M 0s 2024-03-29T04:10:09.834 INFO:teuthology.orchestra.run.smithi156.stderr:158750K .......... .......... .......... .......... .......... 94% 25.8M 0s 2024-03-29T04:10:09.836 INFO:teuthology.orchestra.run.smithi156.stderr:158800K .......... .......... .......... .......... .......... 94% 25.1M 0s 2024-03-29T04:10:09.838 INFO:teuthology.orchestra.run.smithi156.stderr:158850K .......... .......... .......... .......... .......... 94% 25.0M 0s 2024-03-29T04:10:09.840 INFO:teuthology.orchestra.run.smithi156.stderr:158900K .......... .......... .......... .......... .......... 94% 20.9M 0s 2024-03-29T04:10:09.842 INFO:teuthology.orchestra.run.smithi156.stderr:158950K .......... .......... .......... .......... .......... 94% 26.0M 0s 2024-03-29T04:10:09.844 INFO:teuthology.orchestra.run.smithi156.stderr:159000K .......... .......... .......... .......... .......... 94% 25.2M 0s 2024-03-29T04:10:09.846 INFO:teuthology.orchestra.run.smithi156.stderr:159050K .......... .......... .......... .......... .......... 94% 25.3M 0s 2024-03-29T04:10:09.848 INFO:teuthology.orchestra.run.smithi156.stderr:159100K .......... .......... .......... .......... .......... 94% 22.8M 0s 2024-03-29T04:10:09.850 INFO:teuthology.orchestra.run.smithi156.stderr:159150K .......... .......... .......... .......... .......... 94% 29.9M 0s 2024-03-29T04:10:09.851 INFO:teuthology.orchestra.run.smithi156.stderr:159200K .......... .......... .......... .......... .......... 94% 30.4M 0s 2024-03-29T04:10:09.853 INFO:teuthology.orchestra.run.smithi156.stderr:159250K .......... .......... .......... .......... .......... 94% 30.6M 0s 2024-03-29T04:10:09.855 INFO:teuthology.orchestra.run.smithi156.stderr:159300K .......... .......... .......... .......... .......... 94% 25.6M 0s 2024-03-29T04:10:09.857 INFO:teuthology.orchestra.run.smithi156.stderr:159350K .......... .......... .......... .......... .......... 94% 27.4M 0s 2024-03-29T04:10:09.858 INFO:teuthology.orchestra.run.smithi156.stderr:159400K .......... .......... .......... .......... .......... 94% 30.2M 0s 2024-03-29T04:10:09.860 INFO:teuthology.orchestra.run.smithi156.stderr:159450K .......... .......... .......... .......... .......... 94% 28.5M 0s 2024-03-29T04:10:09.862 INFO:teuthology.orchestra.run.smithi156.stderr:159500K .......... .......... .......... .......... .......... 94% 25.8M 0s 2024-03-29T04:10:09.864 INFO:teuthology.orchestra.run.smithi156.stderr:159550K .......... .......... .......... .......... .......... 94% 30.5M 0s 2024-03-29T04:10:09.865 INFO:teuthology.orchestra.run.smithi156.stderr:159600K .......... .......... .......... .......... .......... 94% 30.2M 0s 2024-03-29T04:10:09.867 INFO:teuthology.orchestra.run.smithi156.stderr:159650K .......... .......... .......... .......... .......... 94% 30.4M 0s 2024-03-29T04:10:09.869 INFO:teuthology.orchestra.run.smithi156.stderr:159700K .......... .......... .......... .......... .......... 94% 26.3M 0s 2024-03-29T04:10:09.870 INFO:teuthology.orchestra.run.smithi156.stderr:159750K .......... .......... .......... .......... .......... 94% 33.2M 0s 2024-03-29T04:10:09.872 INFO:teuthology.orchestra.run.smithi156.stderr:159800K .......... .......... .......... .......... .......... 94% 29.7M 0s 2024-03-29T04:10:09.874 INFO:teuthology.orchestra.run.smithi156.stderr:159850K .......... .......... .......... .......... .......... 94% 28.1M 0s 2024-03-29T04:10:09.875 INFO:teuthology.orchestra.run.smithi156.stderr:159900K .......... .......... .......... .......... .......... 94% 26.6M 0s 2024-03-29T04:10:09.877 INFO:teuthology.orchestra.run.smithi156.stderr:159950K .......... .......... .......... .......... .......... 94% 27.0M 0s 2024-03-29T04:10:09.879 INFO:teuthology.orchestra.run.smithi156.stderr:160000K .......... .......... .......... .......... .......... 94% 33.2M 0s 2024-03-29T04:10:09.880 INFO:teuthology.orchestra.run.smithi156.stderr:160050K .......... .......... .......... .......... .......... 94% 38.0M 0s 2024-03-29T04:10:09.882 INFO:teuthology.orchestra.run.smithi156.stderr:160100K .......... .......... .......... .......... .......... 95% 26.2M 0s 2024-03-29T04:10:09.884 INFO:teuthology.orchestra.run.smithi156.stderr:160150K .......... .......... .......... .......... .......... 95% 26.6M 0s 2024-03-29T04:10:09.885 INFO:teuthology.orchestra.run.smithi156.stderr:160200K .......... .......... .......... .......... .......... 95% 28.7M 0s 2024-03-29T04:10:09.887 INFO:teuthology.orchestra.run.smithi156.stderr:160250K .......... .......... .......... .......... .......... 95% 31.5M 0s 2024-03-29T04:10:09.889 INFO:teuthology.orchestra.run.smithi156.stderr:160300K .......... .......... .......... .......... .......... 95% 28.8M 0s 2024-03-29T04:10:09.890 INFO:teuthology.orchestra.run.smithi156.stderr:160350K .......... .......... .......... .......... .......... 95% 30.9M 0s 2024-03-29T04:10:09.892 INFO:teuthology.orchestra.run.smithi156.stderr:160400K .......... .......... .......... .......... .......... 95% 31.7M 0s 2024-03-29T04:10:09.893 INFO:teuthology.orchestra.run.smithi156.stderr:160450K .......... .......... .......... .......... .......... 95% 31.7M 0s 2024-03-29T04:10:09.895 INFO:teuthology.orchestra.run.smithi156.stderr:160500K .......... .......... .......... .......... .......... 95% 26.7M 0s 2024-03-29T04:10:09.897 INFO:teuthology.orchestra.run.smithi156.stderr:160550K .......... .......... .......... .......... .......... 95% 32.4M 0s 2024-03-29T04:10:09.898 INFO:teuthology.orchestra.run.smithi156.stderr:160600K .......... .......... .......... .......... .......... 95% 31.1M 0s 2024-03-29T04:10:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:160650K .......... .......... .......... .......... .......... 95% 32.9M 0s 2024-03-29T04:10:09.901 INFO:teuthology.orchestra.run.smithi156.stderr:160700K .......... .......... .......... .......... .......... 95% 27.6M 0s 2024-03-29T04:10:09.903 INFO:teuthology.orchestra.run.smithi156.stderr:160750K .......... .......... .......... .......... .......... 95% 31.7M 0s 2024-03-29T04:10:09.905 INFO:teuthology.orchestra.run.smithi156.stderr:160800K .......... .......... .......... .......... .......... 95% 31.6M 0s 2024-03-29T04:10:09.906 INFO:teuthology.orchestra.run.smithi156.stderr:160850K .......... .......... .......... .......... .......... 95% 30.9M 0s 2024-03-29T04:10:09.908 INFO:teuthology.orchestra.run.smithi156.stderr:160900K .......... .......... .......... .......... .......... 95% 28.1M 0s 2024-03-29T04:10:09.909 INFO:teuthology.orchestra.run.smithi156.stderr:160950K .......... .......... .......... .......... .......... 95% 32.5M 0s 2024-03-29T04:10:09.911 INFO:teuthology.orchestra.run.smithi156.stderr:161000K .......... .......... .......... .......... .......... 95% 31.9M 0s 2024-03-29T04:10:09.913 INFO:teuthology.orchestra.run.smithi156.stderr:161050K .......... .......... .......... .......... .......... 95% 31.3M 0s 2024-03-29T04:10:09.914 INFO:teuthology.orchestra.run.smithi156.stderr:161100K .......... .......... .......... .......... .......... 95% 27.5M 0s 2024-03-29T04:10:09.916 INFO:teuthology.orchestra.run.smithi156.stderr:161150K .......... .......... .......... .......... .......... 95% 30.8M 0s 2024-03-29T04:10:09.918 INFO:teuthology.orchestra.run.smithi156.stderr:161200K .......... .......... .......... .......... .......... 95% 26.1M 0s 2024-03-29T04:10:09.919 INFO:teuthology.orchestra.run.smithi156.stderr:161250K .......... .......... .......... .......... .......... 95% 31.8M 0s 2024-03-29T04:10:09.921 INFO:teuthology.orchestra.run.smithi156.stderr:161300K .......... .......... .......... .......... .......... 95% 27.8M 0s 2024-03-29T04:10:09.922 INFO:teuthology.orchestra.run.smithi156.stderr:161350K .......... .......... .......... .......... .......... 95% 33.2M 0s 2024-03-29T04:10:09.924 INFO:teuthology.orchestra.run.smithi156.stderr:161400K .......... .......... .......... .......... .......... 95% 32.9M 0s 2024-03-29T04:10:09.926 INFO:teuthology.orchestra.run.smithi156.stderr:161450K .......... .......... .......... .......... .......... 95% 31.1M 0s 2024-03-29T04:10:09.927 INFO:teuthology.orchestra.run.smithi156.stderr:161500K .......... .......... .......... .......... .......... 95% 27.6M 0s 2024-03-29T04:10:09.929 INFO:teuthology.orchestra.run.smithi156.stderr:161550K .......... .......... .......... .......... .......... 95% 30.7M 0s 2024-03-29T04:10:09.931 INFO:teuthology.orchestra.run.smithi156.stderr:161600K .......... .......... .......... .......... .......... 95% 27.7M 0s 2024-03-29T04:10:09.933 INFO:teuthology.orchestra.run.smithi156.stderr:161650K .......... .......... .......... .......... .......... 95% 26.1M 0s 2024-03-29T04:10:09.935 INFO:teuthology.orchestra.run.smithi156.stderr:161700K .......... .......... .......... .......... .......... 95% 23.2M 0s 2024-03-29T04:10:09.936 INFO:teuthology.orchestra.run.smithi156.stderr:161750K .......... .......... .......... .......... .......... 95% 26.7M 0s 2024-03-29T04:10:09.938 INFO:teuthology.orchestra.run.smithi156.stderr:161800K .......... .......... .......... .......... .......... 96% 28.5M 0s 2024-03-29T04:10:09.940 INFO:teuthology.orchestra.run.smithi156.stderr:161850K .......... .......... .......... .......... .......... 96% 28.6M 0s 2024-03-29T04:10:09.942 INFO:teuthology.orchestra.run.smithi156.stderr:161900K .......... .......... .......... .......... .......... 96% 24.5M 0s 2024-03-29T04:10:09.944 INFO:teuthology.orchestra.run.smithi156.stderr:161950K .......... .......... .......... .......... .......... 96% 27.4M 0s 2024-03-29T04:10:09.945 INFO:teuthology.orchestra.run.smithi156.stderr:162000K .......... .......... .......... .......... .......... 96% 27.2M 0s 2024-03-29T04:10:09.947 INFO:teuthology.orchestra.run.smithi156.stderr:162050K .......... .......... .......... .......... .......... 96% 26.2M 0s 2024-03-29T04:10:09.949 INFO:teuthology.orchestra.run.smithi156.stderr:162100K .......... .......... .......... .......... .......... 96% 22.5M 0s 2024-03-29T04:10:09.951 INFO:teuthology.orchestra.run.smithi156.stderr:162150K .......... .......... .......... .......... .......... 96% 24.8M 0s 2024-03-29T04:10:09.953 INFO:teuthology.orchestra.run.smithi156.stderr:162200K .......... .......... .......... .......... .......... 96% 24.7M 0s 2024-03-29T04:10:09.955 INFO:teuthology.orchestra.run.smithi156.stderr:162250K .......... .......... .......... .......... .......... 96% 28.0M 0s 2024-03-29T04:10:09.957 INFO:teuthology.orchestra.run.smithi156.stderr:162300K .......... .......... .......... .......... .......... 96% 23.8M 0s 2024-03-29T04:10:09.959 INFO:teuthology.orchestra.run.smithi156.stderr:162350K .......... .......... .......... .......... .......... 96% 27.1M 0s 2024-03-29T04:10:09.961 INFO:teuthology.orchestra.run.smithi156.stderr:162400K .......... .......... .......... .......... .......... 96% 26.4M 0s 2024-03-29T04:10:09.963 INFO:teuthology.orchestra.run.smithi156.stderr:162450K .......... .......... .......... .......... .......... 96% 26.4M 0s 2024-03-29T04:10:09.965 INFO:teuthology.orchestra.run.smithi156.stderr:162500K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-03-29T04:10:09.967 INFO:teuthology.orchestra.run.smithi156.stderr:162550K .......... .......... .......... .......... .......... 96% 26.4M 0s 2024-03-29T04:10:09.968 INFO:teuthology.orchestra.run.smithi156.stderr:162600K .......... .......... .......... .......... .......... 96% 26.4M 0s 2024-03-29T04:10:09.971 INFO:teuthology.orchestra.run.smithi156.stderr:162650K .......... .......... .......... .......... .......... 96% 24.6M 0s 2024-03-29T04:10:09.973 INFO:teuthology.orchestra.run.smithi156.stderr:162700K .......... .......... .......... .......... .......... 96% 20.8M 0s 2024-03-29T04:10:09.975 INFO:teuthology.orchestra.run.smithi156.stderr:162750K .......... .......... .......... .......... .......... 96% 24.8M 0s 2024-03-29T04:10:09.977 INFO:teuthology.orchestra.run.smithi156.stderr:162800K .......... .......... .......... .......... .......... 96% 24.9M 0s 2024-03-29T04:10:09.979 INFO:teuthology.orchestra.run.smithi156.stderr:162850K .......... .......... .......... .......... .......... 96% 25.7M 0s 2024-03-29T04:10:09.981 INFO:teuthology.orchestra.run.smithi156.stderr:162900K .......... .......... .......... .......... .......... 96% 21.5M 0s 2024-03-29T04:10:09.983 INFO:teuthology.orchestra.run.smithi156.stderr:162950K .......... .......... .......... .......... .......... 96% 26.4M 0s 2024-03-29T04:10:09.985 INFO:teuthology.orchestra.run.smithi156.stderr:163000K .......... .......... .......... .......... .......... 96% 25.9M 0s 2024-03-29T04:10:09.987 INFO:teuthology.orchestra.run.smithi156.stderr:163050K .......... .......... .......... .......... .......... 96% 24.8M 0s 2024-03-29T04:10:09.989 INFO:teuthology.orchestra.run.smithi156.stderr:163100K .......... .......... .......... .......... .......... 96% 21.7M 0s 2024-03-29T04:10:09.991 INFO:teuthology.orchestra.run.smithi156.stderr:163150K .......... .......... .......... .......... .......... 96% 26.0M 0s 2024-03-29T04:10:09.993 INFO:teuthology.orchestra.run.smithi156.stderr:163200K .......... .......... .......... .......... .......... 96% 27.1M 0s 2024-03-29T04:10:09.995 INFO:teuthology.orchestra.run.smithi156.stderr:163250K .......... .......... .......... .......... .......... 96% 24.9M 0s 2024-03-29T04:10:09.997 INFO:teuthology.orchestra.run.smithi156.stderr:163300K .......... .......... .......... .......... .......... 96% 21.5M 0s 2024-03-29T04:10:09.999 INFO:teuthology.orchestra.run.smithi156.stderr:163350K .......... .......... .......... .......... .......... 96% 23.8M 0s 2024-03-29T04:10:10.001 INFO:teuthology.orchestra.run.smithi156.stderr:163400K .......... .......... .......... .......... .......... 96% 25.1M 0s 2024-03-29T04:10:10.007 INFO:teuthology.orchestra.run.smithi156.stderr:163450K .......... .......... .......... .......... .......... 96% 23.6M 0s 2024-03-29T04:10:10.008 INFO:teuthology.orchestra.run.smithi156.stderr:163500K .......... .......... .......... .......... .......... 97% 20.1M 0s 2024-03-29T04:10:10.008 INFO:teuthology.orchestra.run.smithi156.stderr:163550K .......... .......... .......... .......... .......... 97% 21.9M 0s 2024-03-29T04:10:10.010 INFO:teuthology.orchestra.run.smithi156.stderr:163600K .......... .......... .......... .......... .......... 97% 23.3M 0s 2024-03-29T04:10:10.012 INFO:teuthology.orchestra.run.smithi156.stderr:163650K .......... .......... .......... .......... .......... 97% 21.9M 0s 2024-03-29T04:10:10.015 INFO:teuthology.orchestra.run.smithi156.stderr:163700K .......... .......... .......... .......... .......... 97% 17.3M 0s 2024-03-29T04:10:10.017 INFO:teuthology.orchestra.run.smithi156.stderr:163750K .......... .......... .......... .......... .......... 97% 20.4M 0s 2024-03-29T04:10:10.019 INFO:teuthology.orchestra.run.smithi156.stderr:163800K .......... .......... .......... .......... .......... 97% 23.3M 0s 2024-03-29T04:10:10.021 INFO:teuthology.orchestra.run.smithi156.stderr:163850K .......... .......... .......... .......... .......... 97% 29.5M 0s 2024-03-29T04:10:10.023 INFO:teuthology.orchestra.run.smithi156.stderr:163900K .......... .......... .......... .......... .......... 97% 26.8M 0s 2024-03-29T04:10:10.024 INFO:teuthology.orchestra.run.smithi156.stderr:163950K .......... .......... .......... .......... .......... 97% 33.2M 0s 2024-03-29T04:10:10.026 INFO:teuthology.orchestra.run.smithi156.stderr:164000K .......... .......... .......... .......... .......... 97% 31.4M 0s 2024-03-29T04:10:10.027 INFO:teuthology.orchestra.run.smithi156.stderr:164050K .......... .......... .......... .......... .......... 97% 30.9M 0s 2024-03-29T04:10:10.029 INFO:teuthology.orchestra.run.smithi156.stderr:164100K .......... .......... .......... .......... .......... 97% 27.2M 0s 2024-03-29T04:10:10.031 INFO:teuthology.orchestra.run.smithi156.stderr:164150K .......... .......... .......... .......... .......... 97% 31.0M 0s 2024-03-29T04:10:10.032 INFO:teuthology.orchestra.run.smithi156.stderr:164200K .......... .......... .......... .......... .......... 97% 31.4M 0s 2024-03-29T04:10:10.034 INFO:teuthology.orchestra.run.smithi156.stderr:164250K .......... .......... .......... .......... .......... 97% 30.9M 0s 2024-03-29T04:10:10.035 INFO:teuthology.orchestra.run.smithi156.stderr:164300K .......... .......... .......... .......... .......... 97% 27.2M 0s 2024-03-29T04:10:10.037 INFO:teuthology.orchestra.run.smithi156.stderr:164350K .......... .......... .......... .......... .......... 97% 32.1M 0s 2024-03-29T04:10:10.038 INFO:teuthology.orchestra.run.smithi156.stderr:164400K .......... .......... .......... .......... .......... 97% 34.9M 0s 2024-03-29T04:10:10.040 INFO:teuthology.orchestra.run.smithi156.stderr:164450K .......... .......... .......... .......... .......... 97% 35.0M 0s 2024-03-29T04:10:10.042 INFO:teuthology.orchestra.run.smithi156.stderr:164500K .......... .......... .......... .......... .......... 97% 30.0M 0s 2024-03-29T04:10:10.043 INFO:teuthology.orchestra.run.smithi156.stderr:164550K .......... .......... .......... .......... .......... 97% 32.2M 0s 2024-03-29T04:10:10.045 INFO:teuthology.orchestra.run.smithi156.stderr:164600K .......... .......... .......... .......... .......... 97% 30.6M 0s 2024-03-29T04:10:10.046 INFO:teuthology.orchestra.run.smithi156.stderr:164650K .......... .......... .......... .......... .......... 97% 31.4M 0s 2024-03-29T04:10:10.048 INFO:teuthology.orchestra.run.smithi156.stderr:164700K .......... .......... .......... .......... .......... 97% 27.1M 0s 2024-03-29T04:10:10.049 INFO:teuthology.orchestra.run.smithi156.stderr:164750K .......... .......... .......... .......... .......... 97% 32.1M 0s 2024-03-29T04:10:10.051 INFO:teuthology.orchestra.run.smithi156.stderr:164800K .......... .......... .......... .......... .......... 97% 31.1M 0s 2024-03-29T04:10:10.053 INFO:teuthology.orchestra.run.smithi156.stderr:164850K .......... .......... .......... .......... .......... 97% 32.6M 0s 2024-03-29T04:10:10.054 INFO:teuthology.orchestra.run.smithi156.stderr:164900K .......... .......... .......... .......... .......... 97% 27.2M 0s 2024-03-29T04:10:10.056 INFO:teuthology.orchestra.run.smithi156.stderr:164950K .......... .......... .......... .......... .......... 97% 32.2M 0s 2024-03-29T04:10:10.057 INFO:teuthology.orchestra.run.smithi156.stderr:165000K .......... .......... .......... .......... .......... 97% 30.9M 0s 2024-03-29T04:10:10.059 INFO:teuthology.orchestra.run.smithi156.stderr:165050K .......... .......... .......... .......... .......... 97% 30.2M 0s 2024-03-29T04:10:10.061 INFO:teuthology.orchestra.run.smithi156.stderr:165100K .......... .......... .......... .......... .......... 97% 26.1M 0s 2024-03-29T04:10:10.062 INFO:teuthology.orchestra.run.smithi156.stderr:165150K .......... .......... .......... .......... .......... 98% 29.1M 0s 2024-03-29T04:10:10.064 INFO:teuthology.orchestra.run.smithi156.stderr:165200K .......... .......... .......... .......... .......... 98% 30.1M 0s 2024-03-29T04:10:10.066 INFO:teuthology.orchestra.run.smithi156.stderr:165250K .......... .......... .......... .......... .......... 98% 28.4M 0s 2024-03-29T04:10:10.068 INFO:teuthology.orchestra.run.smithi156.stderr:165300K .......... .......... .......... .......... .......... 98% 24.9M 0s 2024-03-29T04:10:10.069 INFO:teuthology.orchestra.run.smithi156.stderr:165350K .......... .......... .......... .......... .......... 98% 29.4M 0s 2024-03-29T04:10:10.071 INFO:teuthology.orchestra.run.smithi156.stderr:165400K .......... .......... .......... .......... .......... 98% 29.5M 0s 2024-03-29T04:10:10.073 INFO:teuthology.orchestra.run.smithi156.stderr:165450K .......... .......... .......... .......... .......... 98% 28.7M 0s 2024-03-29T04:10:10.075 INFO:teuthology.orchestra.run.smithi156.stderr:165500K .......... .......... .......... .......... .......... 98% 25.5M 0s 2024-03-29T04:10:10.077 INFO:teuthology.orchestra.run.smithi156.stderr:165550K .......... .......... .......... .......... .......... 98% 29.8M 0s 2024-03-29T04:10:10.078 INFO:teuthology.orchestra.run.smithi156.stderr:165600K .......... .......... .......... .......... .......... 98% 29.7M 0s 2024-03-29T04:10:10.080 INFO:teuthology.orchestra.run.smithi156.stderr:165650K .......... .......... .......... .......... .......... 98% 29.9M 0s 2024-03-29T04:10:10.082 INFO:teuthology.orchestra.run.smithi156.stderr:165700K .......... .......... .......... .......... .......... 98% 24.7M 0s 2024-03-29T04:10:10.083 INFO:teuthology.orchestra.run.smithi156.stderr:165750K .......... .......... .......... .......... .......... 98% 28.1M 0s 2024-03-29T04:10:10.085 INFO:teuthology.orchestra.run.smithi156.stderr:165800K .......... .......... .......... .......... .......... 98% 28.2M 0s 2024-03-29T04:10:10.087 INFO:teuthology.orchestra.run.smithi156.stderr:165850K .......... .......... .......... .......... .......... 98% 28.0M 0s 2024-03-29T04:10:10.089 INFO:teuthology.orchestra.run.smithi156.stderr:165900K .......... .......... .......... .......... .......... 98% 24.5M 0s 2024-03-29T04:10:10.091 INFO:teuthology.orchestra.run.smithi156.stderr:165950K .......... .......... .......... .......... .......... 98% 29.4M 0s 2024-03-29T04:10:10.092 INFO:teuthology.orchestra.run.smithi156.stderr:166000K .......... .......... .......... .......... .......... 98% 29.0M 0s 2024-03-29T04:10:10.094 INFO:teuthology.orchestra.run.smithi156.stderr:166050K .......... .......... .......... .......... .......... 98% 29.6M 0s 2024-03-29T04:10:10.096 INFO:teuthology.orchestra.run.smithi156.stderr:166100K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-03-29T04:10:10.097 INFO:teuthology.orchestra.run.smithi156.stderr:166150K .......... .......... .......... .......... .......... 98% 29.7M 0s 2024-03-29T04:10:10.100 INFO:teuthology.orchestra.run.smithi156.stderr:166200K .......... .......... .......... .......... .......... 98% 27.4M 0s 2024-03-29T04:10:10.102 INFO:teuthology.orchestra.run.smithi156.stderr:166250K .......... .......... .......... .......... .......... 98% 27.2M 0s 2024-03-29T04:10:10.103 INFO:teuthology.orchestra.run.smithi156.stderr:166300K .......... .......... .......... .......... .......... 98% 22.0M 0s 2024-03-29T04:10:10.105 INFO:teuthology.orchestra.run.smithi156.stderr:166350K .......... .......... .......... .......... .......... 98% 25.4M 0s 2024-03-29T04:10:10.107 INFO:teuthology.orchestra.run.smithi156.stderr:166400K .......... .......... .......... .......... .......... 98% 26.4M 0s 2024-03-29T04:10:10.109 INFO:teuthology.orchestra.run.smithi156.stderr:166450K .......... .......... .......... .......... .......... 98% 27.0M 0s 2024-03-29T04:10:10.111 INFO:teuthology.orchestra.run.smithi156.stderr:166500K .......... .......... .......... .......... .......... 98% 24.3M 0s 2024-03-29T04:10:10.113 INFO:teuthology.orchestra.run.smithi156.stderr:166550K .......... .......... .......... .......... .......... 98% 30.1M 0s 2024-03-29T04:10:10.114 INFO:teuthology.orchestra.run.smithi156.stderr:166600K .......... .......... .......... .......... .......... 98% 28.6M 0s 2024-03-29T04:10:10.116 INFO:teuthology.orchestra.run.smithi156.stderr:166650K .......... .......... .......... .......... .......... 98% 30.0M 0s 2024-03-29T04:10:10.118 INFO:teuthology.orchestra.run.smithi156.stderr:166700K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-03-29T04:10:10.119 INFO:teuthology.orchestra.run.smithi156.stderr:166750K .......... .......... .......... .......... .......... 98% 29.7M 0s 2024-03-29T04:10:10.121 INFO:teuthology.orchestra.run.smithi156.stderr:166800K .......... .......... .......... .......... .......... 98% 29.9M 0s 2024-03-29T04:10:10.123 INFO:teuthology.orchestra.run.smithi156.stderr:166850K .......... .......... .......... .......... .......... 99% 26.8M 0s 2024-03-29T04:10:10.125 INFO:teuthology.orchestra.run.smithi156.stderr:166900K .......... .......... .......... .......... .......... 99% 24.6M 0s 2024-03-29T04:10:10.127 INFO:teuthology.orchestra.run.smithi156.stderr:166950K .......... .......... .......... .......... .......... 99% 31.3M 0s 2024-03-29T04:10:10.128 INFO:teuthology.orchestra.run.smithi156.stderr:167000K .......... .......... .......... .......... .......... 99% 30.5M 0s 2024-03-29T04:10:10.130 INFO:teuthology.orchestra.run.smithi156.stderr:167050K .......... .......... .......... .......... .......... 99% 28.1M 0s 2024-03-29T04:10:10.132 INFO:teuthology.orchestra.run.smithi156.stderr:167100K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-03-29T04:10:10.134 INFO:teuthology.orchestra.run.smithi156.stderr:167150K .......... .......... .......... .......... .......... 99% 27.8M 0s 2024-03-29T04:10:10.135 INFO:teuthology.orchestra.run.smithi156.stderr:167200K .......... .......... .......... .......... .......... 99% 27.4M 0s 2024-03-29T04:10:10.137 INFO:teuthology.orchestra.run.smithi156.stderr:167250K .......... .......... .......... .......... .......... 99% 27.8M 0s 2024-03-29T04:10:10.139 INFO:teuthology.orchestra.run.smithi156.stderr:167300K .......... .......... .......... .......... .......... 99% 23.5M 0s 2024-03-29T04:10:10.141 INFO:teuthology.orchestra.run.smithi156.stderr:167350K .......... .......... .......... .......... .......... 99% 27.9M 0s 2024-03-29T04:10:10.143 INFO:teuthology.orchestra.run.smithi156.stderr:167400K .......... .......... .......... .......... .......... 99% 28.0M 0s 2024-03-29T04:10:10.145 INFO:teuthology.orchestra.run.smithi156.stderr:167450K .......... .......... .......... .......... .......... 99% 26.9M 0s 2024-03-29T04:10:10.147 INFO:teuthology.orchestra.run.smithi156.stderr:167500K .......... .......... .......... .......... .......... 99% 23.2M 0s 2024-03-29T04:10:10.148 INFO:teuthology.orchestra.run.smithi156.stderr:167550K .......... .......... .......... .......... .......... 99% 27.2M 0s 2024-03-29T04:10:10.150 INFO:teuthology.orchestra.run.smithi156.stderr:167600K .......... .......... .......... .......... .......... 99% 26.8M 0s 2024-03-29T04:10:10.152 INFO:teuthology.orchestra.run.smithi156.stderr:167650K .......... .......... .......... .......... .......... 99% 26.2M 0s 2024-03-29T04:10:10.154 INFO:teuthology.orchestra.run.smithi156.stderr:167700K .......... .......... .......... .......... .......... 99% 22.5M 0s 2024-03-29T04:10:10.156 INFO:teuthology.orchestra.run.smithi156.stderr:167750K .......... .......... .......... .......... .......... 99% 27.4M 0s 2024-03-29T04:10:10.158 INFO:teuthology.orchestra.run.smithi156.stderr:167800K .......... .......... .......... .......... .......... 99% 27.7M 0s 2024-03-29T04:10:10.160 INFO:teuthology.orchestra.run.smithi156.stderr:167850K .......... .......... .......... .......... .......... 99% 27.5M 0s 2024-03-29T04:10:10.162 INFO:teuthology.orchestra.run.smithi156.stderr:167900K .......... .......... .......... .......... .......... 99% 21.8M 0s 2024-03-29T04:10:10.163 INFO:teuthology.orchestra.run.smithi156.stderr:167950K .......... .......... .......... .......... .......... 99% 44.2M 0s 2024-03-29T04:10:10.164 INFO:teuthology.orchestra.run.smithi156.stderr:168000K .......... .......... .......... .......... .......... 99% 33.8M 0s 2024-03-29T04:10:10.166 INFO:teuthology.orchestra.run.smithi156.stderr:168050K .......... .......... .......... .......... .......... 99% 33.9M 0s 2024-03-29T04:10:10.168 INFO:teuthology.orchestra.run.smithi156.stderr:168100K .......... .......... .......... .......... .......... 99% 29.6M 0s 2024-03-29T04:10:10.169 INFO:teuthology.orchestra.run.smithi156.stderr:168150K .......... .......... .......... .......... .......... 99% 32.3M 0s 2024-03-29T04:10:10.171 INFO:teuthology.orchestra.run.smithi156.stderr:168200K .......... .......... .......... .......... .......... 99% 29.0M 0s 2024-03-29T04:10:10.172 INFO:teuthology.orchestra.run.smithi156.stderr:168250K .......... .......... .......... .......... .......... 99% 31.2M 0s 2024-03-29T04:10:10.174 INFO:teuthology.orchestra.run.smithi156.stderr:168300K .......... .......... .......... .......... .......... 99% 30.1M 0s 2024-03-29T04:10:10.176 INFO:teuthology.orchestra.run.smithi156.stderr:168350K .......... .......... .......... .......... .......... 99% 29.2M 0s 2024-03-29T04:10:10.179 INFO:teuthology.orchestra.run.smithi156.stderr:168400K .......... .......... .......... .......... .......... 99% 29.2M 0s 2024-03-29T04:10:10.179 INFO:teuthology.orchestra.run.smithi156.stderr:168450K .......... .......... .......... .......... .......... 99% 36.9M 0s 2024-03-29T04:10:10.180 INFO:teuthology.orchestra.run.smithi156.stderr:168500K .......... .......... .......... .......... .......... 99% 33.3M 0s 2024-03-29T04:10:10.181 INFO:teuthology.orchestra.run.smithi156.stderr:168550K .......... .......... . 100% 50.5M=6.0s 2024-03-29T04:10:10.181 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:10:10.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29 04:10:10 (27.5 MB/s) - ‘linux-5.4.tar.gz’ saved [172616875/172616875] 2024-03-29T04:10:10.181 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:10:10.182 INFO:tasks.cephfs.test_uninlining:Running Data Uninlining test with: without clearing cache, without unmount client, without mount.open_background, without multimds, without snapshots 2024-03-29T04:10:10.182 DEBUG:tasks.cephfs.test_uninlining:setting max_mds:1 2024-03-29T04:10:10.183 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-29T04:10:10.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.347+0000 7fb4ceffd640 1 Processor -- start 2024-03-29T04:10:10.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.347+0000 7fb4ceffd640 1 -- start start 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ceffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb4c815e920 0x7fb4c80d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ceffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb4c80cfc30 0x7fb4c80d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ceffd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c80cf190 con 0x7fb4c80d2120 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ceffd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c80cf7a0 con 0x7fb4c815e920 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ceffd640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c80d3080 con 0x7fb4c80cfc30 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ccd99640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb4c815e920 0x7fb4c80d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ccd99640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb4c815e920 0x7fb4c80d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:53840/0 (socket says 172.21.15.156:53840) 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.348+0000 7fb4ccd99640 1 -- 172.21.15.156:0/311294159 learned_addr learned my addr 172.21.15.156:0/311294159 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.349+0000 7fb4cd59a640 1 --2- 172.21.15.156:0/311294159 >> v2:172.21.15.203:3300/0 conn(0x7fb4c80cfc30 0x7fb4c80d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:10.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.349+0000 7fb4ccd99640 1 -- 172.21.15.156:0/311294159 >> v2:172.21.15.203:3300/0 conn(0x7fb4c80cfc30 msgr2=0x7fb4c80d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.349+0000 7fb4ccd99640 1 --2- 172.21.15.156:0/311294159 >> v2:172.21.15.203:3300/0 conn(0x7fb4c80cfc30 0x7fb4c80d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.349+0000 7fb4ccd99640 1 -- 172.21.15.156:0/311294159 >> v1:172.21.15.203:6789/0 conn(0x7fb4c80d2120 legacy=0x7fb4c806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.349+0000 7fb4ccd99640 1 -- 172.21.15.156:0/311294159 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c80cf430 con 0x7fb4c815e920 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ccd99640 1 --2- 172.21.15.156:0/311294159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c815e920 0x7fb4c80d26e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb4c808ca00 tx=0x7fb4c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255d5b6d92c61fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4bf7fe640 1 -- 172.21.15.156:0/311294159 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4034070 con 0x7fb4c815e920 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4bf7fe640 1 -- 172.21.15.156:0/311294159 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c4030860 con 0x7fb4c815e920 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4bf7fe640 1 -- 172.21.15.156:0/311294159 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4030c30 con 0x7fb4c815e920 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 -- 172.21.15.156:0/311294159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c815e920 msgr2=0x7fb4c80d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/311294159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c815e920 0x7fb4c80d26e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb4c808ca00 tx=0x7fb4c402b9a0 comp rx=0 tx=0).stop 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 -- 172.21.15.156:0/311294159 shutdown_connections 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/311294159 >> v2:172.21.15.203:3300/0 conn(0x7fb4c80cfc30 0x7fb4c80d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:10.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/311294159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c815e920 0x7fb4c80d26e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:10.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 -- 172.21.15.156:0/311294159 >> 172.21.15.156:0/311294159 conn(0x7fb4c809d7a0 msgr2=0x7fb4c80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:10.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 -- 172.21.15.156:0/311294159 shutdown_connections 2024-03-29T04:10:10.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.350+0000 7fb4ceffd640 1 -- 172.21.15.156:0/311294159 wait complete. 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.351+0000 7fb4ceffd640 1 Processor -- start 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 -- start start 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 0x7fb4c80c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c80d2120 0x7fb4c81e91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb4c815e920 0x7fb4c81ea060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c80d34e0 con 0x7fb4c80cfc30 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c80d3360 con 0x7fb4c80d2120 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ceffd640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c80d2c20 con 0x7fb4c815e920 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.352+0000 7fb4ccd99640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 0x7fb4c80c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:10.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 0x7fb4c80c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:53846/0 (socket says 172.21.15.156:53846) 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 -- 172.21.15.156:0/1091306104 learned_addr learned my addr 172.21.15.156:0/1091306104 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4bffff640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c80d2120 0x7fb4c81e91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4cd59a640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb4c815e920 0x7fb4c81ea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 -- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb4c815e920 msgr2=0x7fb4c81ea060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb4c815e920 0x7fb4c81ea060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 -- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c80d2120 msgr2=0x7fb4c81e91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c80d2120 0x7fb4c81e91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 -- 172.21.15.156:0/1091306104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c80ce160 con 0x7fb4c80cfc30 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ccd99640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 0x7fb4c80c9f60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402b9e0 tx=0x7fb4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4034040 con 0x7fb4c80cfc30 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c4039d90 con 0x7fb4c80cfc30 2024-03-29T04:10:10.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4043610 con 0x7fb4c80cfc30 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c80cc060 con 0x7fb4c80cfc30 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.353+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c80ca4a0 con 0x7fb4c80cfc30 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.354+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c80a2480 con 0x7fb4c80cfc30 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.355+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb4c404a060 con 0x7fb4c80cfc30 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.355+0000 7fb4bd7fa640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb4b406c560 0x7fb4b406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.355+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fb4c40ad7a0 con 0x7fb4c80cfc30 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.355+0000 7fb4bffff640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb4b406c560 0x7fb4b406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:10.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.356+0000 7fb4bffff640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb4b406c560 0x7fb4b406ea20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb4c81f7bf0 tx=0x7fb4b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:10.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.360+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb4c403e070 con 0x7fb4c80cfc30 2024-03-29T04:10:10.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:10.695+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7fb4c804f9d0 con 0x7fb4c80cfc30 2024-03-29T04:10:11.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.069+0000 7fb4bd7fa640 1 -- 172.21.15.156:0/1091306104 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7fb4c4037360 con 0x7fb4c80cfc30 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb4b406c560 msgr2=0x7fb4b406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb4b406c560 0x7fb4b406ea20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb4c81f7bf0 tx=0x7fb4b8009040 comp rx=0 tx=0).stop 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 msgr2=0x7fb4c80c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 0x7fb4c80c9f60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402b9e0 tx=0x7fb4c40046c0 comp rx=0 tx=0).stop 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 shutdown_connections 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb4b406c560 0x7fb4b406ea20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb4c815e920 0x7fb4c81ea060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c80d2120 0x7fb4c81e91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 --2- 172.21.15.156:0/1091306104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4c80cfc30 0x7fb4c80c9f60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 >> 172.21.15.156:0/1091306104 conn(0x7fb4c809d7a0 msgr2=0x7fb4c80b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 shutdown_connections 2024-03-29T04:10:11.051 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:11.071+0000 7fb4ceffd640 1 -- 172.21.15.156:0/1091306104 wait complete. 2024-03-29T04:10:11.061 DEBUG:tasks.cephfs.test_uninlining:getting client_max_inline_size 2024-03-29T04:10:11.061 DEBUG:tasks.cephfs.test_uninlining:got client_max_inline_size:4096 2024-03-29T04:10:11.061 DEBUG:tasks.cephfs.test_uninlining:extracting tarball 2024-03-29T04:10:11.061 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && tar -x -z -f linux-5.4.tar.gz linux-5.4/fs/ceph linux-5.4/fs/orangefs linux-5.4/fs/ext2 2024-03-29T04:10:16.437 DEBUG:tasks.cephfs.test_uninlining:fetching dir list 2024-03-29T04:10:16.437 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && find linux-5.4/ -mindepth 2 -maxdepth 2 -type d 2024-03-29T04:10:16.473 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ceph 2024-03-29T04:10:16.474 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2 2024-03-29T04:10:16.474 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs 2024-03-29T04:10:16.474 DEBUG:tasks.cephfs.test_uninlining:distributing dir pins 2024-03-29T04:10:16.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:16.474 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 linux-5.4/fs/ceph) 2024-03-29T04:10:16.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:16.552 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 linux-5.4/fs/ext2) 2024-03-29T04:10:16.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:16.609 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 linux-5.4/fs/orangefs) 2024-03-29T04:10:16.666 DEBUG:tasks.cephfs.test_uninlining:listing files with size <= 4096 2024-03-29T04:10:16.666 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && find linux-5.4/ -type f -size -4097c 2024-03-29T04:10:16.754 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ceph/Kconfig 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ceph/Makefile 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ceph/ceph_frag.c 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ceph/io.h 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ceph/ioctl.h 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/Kconfig 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/xattr_user.c 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/xattr_security.c 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/Makefile 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/symlink.c 2024-03-29T04:10:16.755 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/acl.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/xattr_trusted.c 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/ext2/xattr.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/Kconfig 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/orangefs-debug.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/orangefs-sysfs.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/dcache.c 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/orangefs-bufmap.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/orangefs-debugfs.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/orangefs-dev-proto.h 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/Makefile 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/symlink.c 2024-03-29T04:10:16.756 INFO:teuthology.orchestra.run.smithi156.stdout:linux-5.4/fs/orangefs/downcall.h 2024-03-29T04:10:16.756 INFO:tasks.cephfs.test_uninlining:Found 23 inlined files 2024-03-29T04:10:16.757 DEBUG:tasks.cephfs.test_uninlining:starting scrub 2024-03-29T04:10:16.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:16.758 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub start / recursive 2024-03-29T04:10:16.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.910+0000 7f859107e640 1 Processor -- start 2024-03-29T04:10:16.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.911+0000 7f859107e640 1 -- start start 2024-03-29T04:10:16.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.911+0000 7f859107e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f858c15ab50 0x7f858c15f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.911+0000 7f859107e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 0x7f858c0725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.911+0000 7f859107e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858c06d9c0 con 0x7f858c0b4d50 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.911+0000 7f859107e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f858c15af30 con 0x7f858c15ab50 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.911+0000 7f859107e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f858c06dca0 con 0x7f858c072160 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858b59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 0x7f858c0725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f858c15ab50 0x7f858c15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858a59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f858c0b4d50 0x7f858c0ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:42716/0 (socket says 172.21.15.156:42716) 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f858c15ab50 0x7f858c15f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:36582/0 (socket says 172.21.15.156:36582) 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858b59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 0x7f858c0725f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:33872/0 (socket says 172.21.15.156:33872) 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858a59c640 1 -- 172.21.15.156:0/2580616434 learned_addr learned my addr 172.21.15.156:0/2580616434 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858ad9d640 1 -- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 msgr2=0x7f858c0725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858ad9d640 1 --2- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 0x7f858c0725f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858ad9d640 1 -- 172.21.15.156:0/2580616434 >> v1:172.21.15.203:6789/0 conn(0x7f858c0b4d50 legacy=0x7f858c0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858ad9d640 1 -- 172.21.15.156:0/2580616434 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858c06df80 con 0x7f858c15ab50 2024-03-29T04:10:16.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.912+0000 7f858b59e640 1 --2- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 0x7f858c0725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.913+0000 7f858ad9d640 1 --2- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c15ab50 0x7f858c15f080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85800097a0 tx=0x7f858002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa788951f51f63c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.913+0000 7f8589d9b640 1 -- 172.21.15.156:0/2580616434 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580034070 con 0x7f858c15ab50 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.913+0000 7f8589d9b640 1 -- 172.21.15.156:0/2580616434 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8580030610 con 0x7f858c15ab50 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.913+0000 7f8589d9b640 1 -- 172.21.15.156:0/2580616434 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85800309e0 con 0x7f858c15ab50 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 -- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c15ab50 msgr2=0x7f858c15f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 --2- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c15ab50 0x7f858c15f080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85800097a0 tx=0x7f858002b960 comp rx=0 tx=0).stop 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 -- 172.21.15.156:0/2580616434 shutdown_connections 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 --2- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c072160 0x7f858c0725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 --2- 172.21.15.156:0/2580616434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c15ab50 0x7f858c15f080 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 -- 172.21.15.156:0/2580616434 >> 172.21.15.156:0/2580616434 conn(0x7f858c0b21e0 msgr2=0x7f858c0b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:16.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 -- 172.21.15.156:0/2580616434 shutdown_connections 2024-03-29T04:10:16.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.914+0000 7f859107e640 1 -- 172.21.15.156:0/2580616434 wait complete. 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.915+0000 7f859107e640 1 Processor -- start 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.915+0000 7f859107e640 1 -- start start 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f859107e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 0x7f858c1f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f859107e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c0b4d50 0x7f858c14d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f859107e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c15ab50 0x7f858c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f859107e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f858c0ad970 con 0x7f858c072160 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f859107e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f858c0b3510 con 0x7f858c0b4d50 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f859107e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f858c0d0470 con 0x7f858c15ab50 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 0x7f858c1f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 0x7f858c1f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42470/0 (socket says 172.21.15.156:42470) 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 -- 172.21.15.156:0/1589756098 learned_addr learned my addr 172.21.15.156:0/1589756098 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858b59e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c15ab50 0x7f858c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858a59c640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c0b4d50 0x7f858c14d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 -- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c15ab50 msgr2=0x7f858c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c15ab50 0x7f858c0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 -- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c0b4d50 msgr2=0x7f858c14d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c0b4d50 0x7f858c14d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858ad9d640 1 -- 172.21.15.156:0/1589756098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858c159d70 con 0x7f858c072160 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.916+0000 7f858a59c640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c0b4d50 0x7f858c14d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.917+0000 7f858ad9d640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 0x7f858c1f3f70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f85800098c0 tx=0x7f85800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.917+0000 7f85737fe640 1 -- 172.21.15.156:0/1589756098 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580034040 con 0x7f858c072160 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.917+0000 7f85737fe640 1 -- 172.21.15.156:0/1589756098 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858003aaa0 con 0x7f858c072160 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.917+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f858c0ccbe0 con 0x7f858c072160 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.917+0000 7f85737fe640 1 -- 172.21.15.156:0/1589756098 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85800443f0 con 0x7f858c072160 2024-03-29T04:10:16.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.917+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f858c14db20 con 0x7f858c072160 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.919+0000 7f85737fe640 1 -- 172.21.15.156:0/1589756098 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f858004b060 con 0x7f858c072160 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.919+0000 7f85737fe640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f857806c530 0x7f857806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.919+0000 7f85737fe640 1 -- 172.21.15.156:0/1589756098 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f85800ad4d0 con 0x7f858c072160 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.919+0000 7f858a59c640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f857806c530 0x7f857806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.920+0000 7f8593c5f740 1 Processor -- start 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.920+0000 7f858a59c640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f857806c530 0x7f857806e9f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f858c0b4c90 tx=0x7f857c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.902 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.920+0000 7f8593c5f740 1 -- start start 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a1359ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a13898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a135a000 0x5564a1389df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8593c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5564a12c3410 con 0x5564a1386b20 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564a1261a80 con 0x5564a135a000 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x5564a12bf3c0 con 0x5564a1386f00 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a1359ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a1359ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42484/0 (socket says 172.21.15.156:42484) 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 -- 172.21.15.156:0/522720947 learned_addr learned my addr 172.21.15.156:0/522720947 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858a59c640 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a13898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858b59e640 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a135a000 0x5564a1389df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 -- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 msgr2=0x5564a13898b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a13898b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 -- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a135a000 msgr2=0x5564a1389df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a135a000 0x5564a1389df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 -- 172.21.15.156:0/522720947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a10a37b0 con 0x5564a1386b20 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858a59c640 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a13898b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f858ad9d640 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a1359ac0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f85800aba20 tx=0x7f858007acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8178ed074e41695f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8570ff9640 1 -- 172.21.15.156:0/522720947 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858007ba70 con 0x5564a1386b20 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8570ff9640 1 -- 172.21.15.156:0/522720947 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858007b370 con 0x5564a1386b20 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.921+0000 7f8570ff9640 1 -- 172.21.15.156:0/522720947 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858007f720 con 0x5564a1386b20 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 -- 172.21.15.156:0/522720947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 msgr2=0x5564a1359ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a1359ac0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f85800aba20 tx=0x7f858007acd0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 -- 172.21.15.156:0/522720947 shutdown_connections 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a135a000 0x5564a1389df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a13898b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 --2- 172.21.15.156:0/522720947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a1359ac0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 -- 172.21.15.156:0/522720947 >> 172.21.15.156:0/522720947 conn(0x5564a135bd30 msgr2=0x5564a13a48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 -- 172.21.15.156:0/522720947 shutdown_connections 2024-03-29T04:10:16.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.922+0000 7f8593c5f740 1 -- 172.21.15.156:0/522720947 wait complete. 2024-03-29T04:10:16.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 -- start start 2024-03-29T04:10:16.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 Processor -- start 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386b20 0x5564a13747f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a138abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 0x5564a138c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5564a1261a80 con 0x5564a13a5110 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564a125f740 con 0x5564a1386b20 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.923+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x5564a12c3410 con 0x5564a1386f00 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8593c5f740 10 client.0 _ll_register_callbacks cb 0x5564a126cb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 0x5564a138c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 0x5564a138c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42496/0 (socket says 172.21.15.156:42496) 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 -- 172.21.15.156:0/3032078552 learned_addr learned my addr 172.21.15.156:0/3032078552 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858ad9d640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386b20 0x5564a13747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858a59c640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a138abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 -- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 msgr2=0x5564a138abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a138abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 -- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386b20 msgr2=0x5564a13747f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386b20 0x5564a13747f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 -- 172.21.15.156:0/3032078552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a1103f40 con 0x5564a13a5110 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f858b59e640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 0x5564a138c740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8584007dd0 tx=0x7f8584012610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad6e7503a7f7924 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8569ffb640 1 -- 172.21.15.156:0/3032078552 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858401c020 con 0x5564a13a5110 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8569ffb640 1 -- 172.21.15.156:0/3032078552 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f8584018670 con 0x5564a13a5110 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f858007b370 con 0x5564a13a5110 2024-03-29T04:10:16.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8569ffb640 1 -- 172.21.15.156:0/3032078552 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f8584026050 con 0x5564a13a5110 2024-03-29T04:10:16.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8593c5f740 10 client.5556 fetch_fsmap learned FSMap version 17 2024-03-29T04:10:16.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5564a0f970a0 con 0x5564a13a5110 2024-03-29T04:10:16.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.924+0000 7f8569ffb640 1 -- 172.21.15.156:0/3032078552 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858007b370 con 0x5564a13a5110 2024-03-29T04:10:16.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8569ffb640 1 -- 172.21.15.156:0/3032078552 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f8584029020 con 0x5564a13a5110 2024-03-29T04:10:16.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8593c5f740 10 client.5556 fetch_fsmap finished waiting for FSMap version 17 2024-03-29T04:10:16.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8593c5f740 10 client.5556 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-29T04:10:16.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8593c5f740 20 client.5556 populate_metadata read hostname 'smithi156' 2024-03-29T04:10:16.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1387400 0x5564a1387880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8593c5f740 4 client.5556 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T04:10:16.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564a12d0950 con 0x5564a1387400 2024-03-29T04:10:16.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f858ad9d640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1387400 0x5564a1387880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f8569ffb640 10 client.5556 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:16.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.925+0000 7f858ad9d640 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1387400 0x5564a1387880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0ec8400abed2c50 server_cookie=5035b25529c86471 in_seq=0 out_seq=0 2024-03-29T04:10:16.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8569ffb640 1 -- 172.21.15.156:0/3032078552 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23239 (crc 0 0 0) 0x5564a12d0950 con 0x5564a1387400 2024-03-29T04:10:16.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8569ffb640 10 client.5556 handle_command_reply: tid=0 2024-03-29T04:10:16.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 client.5556 shutdown 2024-03-29T04:10:16.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 msgr2=0x5564a138c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 0x5564a138c740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8584007dd0 tx=0x7f8584012610 comp rx=0 tx=0).stop 2024-03-29T04:10:16.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 shutdown_connections 2024-03-29T04:10:16.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1387400 0x5564a1387880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a13a5110 0x5564a138c740 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386f00 0x5564a138abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8569ffb640 0 client.5556 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:16.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3032078552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386b20 0x5564a13747f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 >> 172.21.15.156:0/3032078552 conn(0x5564a135a000 msgr2=0x5564a13a6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:16.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 shutdown_connections 2024-03-29T04:10:16.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 1 -- 172.21.15.156:0/3032078552 wait complete. 2024-03-29T04:10:16.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.926+0000 7f8593c5f740 20 client.5556 trim_cache size 0 max 16384 2024-03-29T04:10:16.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.944+0000 7f8593c5f740 1 Processor -- start 2024-03-29T04:10:16.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.944+0000 7f8593c5f740 1 -- start start 2024-03-29T04:10:16.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a13c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386f00 0x5564a1381a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a13a5110 0x5564a1381fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f8593c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f858c200ad0 con 0x5564a1386b20 2024-03-29T04:10:16.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f858c0ca370 con 0x5564a1386f00 2024-03-29T04:10:16.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f858c0c96d0 con 0x5564a13a5110 2024-03-29T04:10:16.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a13c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a13c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42498/0 (socket says 172.21.15.156:42498) 2024-03-29T04:10:16.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 -- 172.21.15.156:0/3510408478 learned_addr learned my addr 172.21.15.156:0/3510408478 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:16.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858a59c640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386f00 0x5564a1381a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858b59e640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a13a5110 0x5564a1381fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 -- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a13a5110 msgr2=0x5564a1381fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a13a5110 0x5564a1381fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 -- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386f00 msgr2=0x5564a1381a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386f00 0x5564a1381a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 -- 172.21.15.156:0/3510408478 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a1360b40 con 0x5564a1386b20 2024-03-29T04:10:16.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858a59c640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386f00 0x5564a1381a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:16.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858b59e640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a13a5110 0x5564a1381fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:16.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f858ad9d640 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a13c74c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8580077780 tx=0x7f8580045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee712297516b4d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f856bfff640 1 -- 172.21.15.156:0/3510408478 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580098040 con 0x5564a1386b20 2024-03-29T04:10:16.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f856bfff640 1 -- 172.21.15.156:0/3510408478 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8580083f50 con 0x5564a1386b20 2024-03-29T04:10:16.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.945+0000 7f856bfff640 1 -- 172.21.15.156:0/3510408478 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858007f720 con 0x5564a1386b20 2024-03-29T04:10:16.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 -- 172.21.15.156:0/3510408478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 msgr2=0x5564a13c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a13c74c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8580077780 tx=0x7f8580045000 comp rx=0 tx=0).stop 2024-03-29T04:10:16.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 -- 172.21.15.156:0/3510408478 shutdown_connections 2024-03-29T04:10:16.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a13a5110 0x5564a1381fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a1386f00 0x5564a1381a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 --2- 172.21.15.156:0/3510408478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386b20 0x5564a13c74c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 -- 172.21.15.156:0/3510408478 >> 172.21.15.156:0/3510408478 conn(0x5564a135a000 msgr2=0x5564a135d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:16.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 -- 172.21.15.156:0/3510408478 shutdown_connections 2024-03-29T04:10:16.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 -- 172.21.15.156:0/3510408478 wait complete. 2024-03-29T04:10:16.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 -- start start 2024-03-29T04:10:16.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.946+0000 7f8593c5f740 1 Processor -- start 2024-03-29T04:10:16.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386b20 0x5564a1373390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 0x5564a1420830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a13a5110 0x5564a13b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5564a134c910 con 0x5564a1386f00 2024-03-29T04:10:16.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f858c0c9d20 con 0x5564a13a5110 2024-03-29T04:10:16.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x5564a12d14b0 con 0x5564a1386b20 2024-03-29T04:10:16.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f8593c5f740 10 client.0 _ll_register_callbacks cb 0x7f8584009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:10:16.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f858a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 0x5564a1420830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f858a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 0x5564a1420830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42504/0 (socket says 172.21.15.156:42504) 2024-03-29T04:10:16.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f858a59c640 1 -- 172.21.15.156:0/2170082979 learned_addr learned my addr 172.21.15.156:0/2170082979 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:16.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f858ad9d640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386b20 0x5564a1373390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.947+0000 7f858b59e640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a13a5110 0x5564a13b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f858a59c640 1 -- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386b20 msgr2=0x5564a1373390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f858a59c640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386b20 0x5564a1373390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f858a59c640 1 -- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a13a5110 msgr2=0x5564a13b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:16.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f858a59c640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a13a5110 0x5564a13b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f858a59c640 1 -- 172.21.15.156:0/2170082979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a1409740 con 0x5564a1386f00 2024-03-29T04:10:16.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f858a59c640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 0x5564a1420830 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f858c0ad490 tx=0x7f857c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be690dc0cc8df0d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:16.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:10:16.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8569ffb640 1 -- 172.21.15.156:0/2170082979 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c029060 con 0x5564a1386f00 2024-03-29T04:10:16.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8569ffb640 1 -- 172.21.15.156:0/2170082979 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f857c010040 con 0x5564a1386f00 2024-03-29T04:10:16.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8580083f50 con 0x5564a1386f00 2024-03-29T04:10:16.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8569ffb640 1 -- 172.21.15.156:0/2170082979 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f857c02f050 con 0x5564a1386f00 2024-03-29T04:10:16.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 10 client.5562 fetch_fsmap learned FSMap version 17 2024-03-29T04:10:16.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5564a1374040 con 0x5564a1386f00 2024-03-29T04:10:16.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8569ffb640 1 -- 172.21.15.156:0/2170082979 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8580083f50 con 0x5564a1386f00 2024-03-29T04:10:16.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8569ffb640 1 -- 172.21.15.156:0/2170082979 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f857c032020 con 0x5564a1386f00 2024-03-29T04:10:16.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 10 client.5562 fetch_fsmap finished waiting for FSMap version 17 2024-03-29T04:10:16.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 10 client.5562 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-29T04:10:16.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 20 client.5562 populate_metadata read hostname 'smithi156' 2024-03-29T04:10:16.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1355280 0x5564a1388360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:16.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 4 client.5562 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-03-29T04:10:16.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.948+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) -- 0x5564a138f7e0 con 0x5564a1355280 2024-03-29T04:10:16.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.949+0000 7f858ad9d640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1355280 0x5564a1388360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:16.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.949+0000 7f8569ffb640 10 client.5562 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:16.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.949+0000 7f858ad9d640 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1355280 0x5564a1388360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64c7890abccfac27 server_cookie=87236f74b2492b56 in_seq=0 out_seq=0 2024-03-29T04:10:16.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8569ffb640 1 -- 172.21.15.156:0/2170082979 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5564a138f7e0 con 0x5564a1355280 2024-03-29T04:10:16.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8569ffb640 10 client.5562 handle_command_reply: tid=0 2024-03-29T04:10:16.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 client.5562 shutdown 2024-03-29T04:10:16.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 msgr2=0x5564a1420830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 0x5564a1420830 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f858c0ad490 tx=0x7f857c000bc0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 shutdown_connections 2024-03-29T04:10:16.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x5564a1355280 0x5564a1388360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564a13a5110 0x5564a13b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564a1386f00 0x5564a1420830 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 --2- 172.21.15.156:0/2170082979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x5564a1386b20 0x5564a1373390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 >> 172.21.15.156:0/2170082979 conn(0x5564a135a000 msgr2=0x5564a1358ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:16.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8569ffb640 0 client.5562 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:16.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 shutdown_connections 2024-03-29T04:10:16.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.950+0000 7f8593c5f740 1 -- 172.21.15.156:0/2170082979 wait complete. 2024-03-29T04:10:16.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f8593c5f740 20 client.5562 trim_cache size 0 max 16384 2024-03-29T04:10:16.952 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:10:16.952 INFO:teuthology.orchestra.run.smithi156.stdout: "return_code": 0, 2024-03-29T04:10:16.952 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_tag": "d3d9bb08-77ad-4349-b350-47b4ffcee287", 2024-03-29T04:10:16.952 INFO:teuthology.orchestra.run.smithi156.stdout: "mode": "asynchronous" 2024-03-29T04:10:16.952 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:10:16.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f857806c530 msgr2=0x7f857806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f857806c530 0x7f857806e9f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f858c0b4c90 tx=0x7f857c009040 comp rx=0 tx=0).stop 2024-03-29T04:10:16.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 msgr2=0x7f858c1f3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:16.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 0x7f858c1f3f70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f85800098c0 tx=0x7f85800046c0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 shutdown_connections 2024-03-29T04:10:16.956 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f857806c530 0x7f857806e9f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f858c15ab50 0x7f858c0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f858c0b4d50 0x7f858c14d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 --2- 172.21.15.156:0/1589756098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f858c072160 0x7f858c1f3f70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:16.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 >> 172.21.15.156:0/1589756098 conn(0x7f858c0b21e0 msgr2=0x7f858c074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:16.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.951+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 shutdown_connections 2024-03-29T04:10:16.957 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:16.952+0000 7f859107e640 1 -- 172.21.15.156:0/1589756098 wait complete. 2024-03-29T04:10:16.958 DEBUG:tasks.cephfs.test_uninlining:scrub start response: {'return_code': 0, 'scrub_tag': 'd3d9bb08-77ad-4349-b350-47b4ffcee287', 'mode': 'asynchronous'} 2024-03-29T04:10:16.958 DEBUG:tasks.cephfs.test_uninlining:waiting for scrub to complete 2024-03-29T04:10:16.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:16.959 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-03-29T04:10:17.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.129+0000 7f2264e1b640 1 Processor -- start 2024-03-29T04:10:17.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.129+0000 7f2264e1b640 1 -- start start 2024-03-29T04:10:17.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f2264e1b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f22600cfc30 0x7f22600d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f2264e1b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22600d2df0 con 0x7f22600d2120 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f2264e1b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22600cf190 con 0x7f226015e920 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f2264e1b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f22600cf7a0 con 0x7f22600cfc30 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f225ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f22600cfc30 0x7f22600d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f225ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f22600cfc30 0x7f22600d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42512/0 (socket says 172.21.15.156:42512) 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f225ed9d640 1 -- 172.21.15.156:0/1275546244 learned_addr learned my addr 172.21.15.156:0/1275546244 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f225ed9d640 1 -- 172.21.15.156:0/1275546244 >> v1:172.21.15.203:6789/0 conn(0x7f226015e920 legacy=0x7f22600d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f225ed9d640 1 -- 172.21.15.156:0/1275546244 >> v1:172.21.15.156:6789/0 conn(0x7f22600d2120 legacy=0x7f226006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.130+0000 7f225ed9d640 1 -- 172.21.15.156:0/1275546244 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22600d3080 con 0x7f22600cfc30 2024-03-29T04:10:17.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.132+0000 7f225ed9d640 1 --2- 172.21.15.156:0/1275546244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22600d00c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2258009e50 tx=0x7f225802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e98e2a5d42d397e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.132+0000 7f225d59a640 1 -- 172.21.15.156:0/1275546244 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225802cba0 con 0x7f22600cfc30 2024-03-29T04:10:17.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.132+0000 7f225d59a640 1 -- 172.21.15.156:0/1275546244 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225802cd40 con 0x7f22600cfc30 2024-03-29T04:10:17.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.132+0000 7f225d59a640 1 -- 172.21.15.156:0/1275546244 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2258034610 con 0x7f22600cfc30 2024-03-29T04:10:17.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 -- 172.21.15.156:0/1275546244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 msgr2=0x7f22600d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 --2- 172.21.15.156:0/1275546244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22600d00c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2258009e50 tx=0x7f225802c000 comp rx=0 tx=0).stop 2024-03-29T04:10:17.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 -- 172.21.15.156:0/1275546244 shutdown_connections 2024-03-29T04:10:17.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 --2- 172.21.15.156:0/1275546244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22600d00c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 -- 172.21.15.156:0/1275546244 >> 172.21.15.156:0/1275546244 conn(0x7f226009d7a0 msgr2=0x7f22600d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:17.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 -- 172.21.15.156:0/1275546244 shutdown_connections 2024-03-29T04:10:17.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.133+0000 7f2264e1b640 1 -- 172.21.15.156:0/1275546244 wait complete. 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.135+0000 7f2264e1b640 1 Processor -- start 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.135+0000 7f2264e1b640 1 -- start start 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f2264e1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22601e8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f2264e1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22600d2120 0x7f22601615a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f2264e1b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f226015e920 0x7f22601f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f2264e1b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f226006cce0 con 0x7f22600cfc30 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f2264e1b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22600d3360 con 0x7f22600d2120 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f2264e1b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2260161110 con 0x7f226015e920 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.136+0000 7f225e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22601e8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22601e8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42520/0 (socket says 172.21.15.156:42520) 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 -- 172.21.15.156:0/2970071275 learned_addr learned my addr 172.21.15.156:0/2970071275 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225ed9d640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f226015e920 0x7f22601f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.117 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225dd9b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22600d2120 0x7f22601615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 -- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f226015e920 msgr2=0x7f22601f84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f226015e920 0x7f22601f84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 -- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22600d2120 msgr2=0x7f22601615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22600d2120 0x7f22601615a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 -- 172.21.15.156:0/2970071275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22601e9290 con 0x7f22600cfc30 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f225e59c640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22601e8d50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f225000c400 tx=0x7f225000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f224effd640 1 -- 172.21.15.156:0/2970071275 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22500052d0 con 0x7f22600cfc30 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f224effd640 1 -- 172.21.15.156:0/2970071275 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2250005490 con 0x7f22600cfc30 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.137+0000 7f224effd640 1 -- 172.21.15.156:0/2970071275 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250007260 con 0x7f22600cfc30 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.138+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2260160500 con 0x7f22600cfc30 2024-03-29T04:10:17.118 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.138+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2260160ac0 con 0x7f22600cfc30 2024-03-29T04:10:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.140+0000 7f224effd640 1 -- 172.21.15.156:0/2970071275 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2250007400 con 0x7f22600cfc30 2024-03-29T04:10:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.140+0000 7f224effd640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f224006c530 0x7f224006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.141+0000 7f224effd640 1 -- 172.21.15.156:0/2970071275 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f225008ac00 con 0x7f22600cfc30 2024-03-29T04:10:17.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.141+0000 7f2267a08740 1 Processor -- start 2024-03-29T04:10:17.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.141+0000 7f225dd9b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f224006c530 0x7f224006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.120 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.141+0000 7f2267a08740 1 -- start start 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.141+0000 7f225dd9b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f224006c530 0x7f224006e9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f22600b4a60 tx=0x7f2254009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f2267a08740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 0x556ffd40f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd43ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd40f6e0 0x556ffd43f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f2267a08740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556ffd379880 con 0x556ffd43c200 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556ffd3218f0 con 0x556ffd43c5e0 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x556ffd375830 con 0x556ffd40f6e0 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 0x556ffd40f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225ed9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd40f6e0 0x556ffd43f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 0x556ffd40f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42522/0 (socket says 172.21.15.156:42522) 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225dd9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd43ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 -- 172.21.15.156:0/1154788656 learned_addr learned my addr 172.21.15.156:0/1154788656 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 -- 172.21.15.156:0/1154788656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd40f6e0 msgr2=0x556ffd43f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd40f6e0 0x556ffd43f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 -- 172.21.15.156:0/1154788656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 msgr2=0x556ffd43ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd43ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.142+0000 7f225e59c640 1 -- 172.21.15.156:0/1154788656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ffd32c140 con 0x556ffd43c200 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.143+0000 7f225e59c640 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 0x556ffd40f1a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2250061130 tx=0x7f225000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6176145198c3b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.143+0000 7f223ffff640 1 -- 172.21.15.156:0/1154788656 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22500551a0 con 0x556ffd43c200 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.143+0000 7f223ffff640 1 -- 172.21.15.156:0/1154788656 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225005f070 con 0x556ffd43c200 2024-03-29T04:10:17.123 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.143+0000 7f223ffff640 1 -- 172.21.15.156:0/1154788656 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250057f80 con 0x556ffd43c200 2024-03-29T04:10:17.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 -- 172.21.15.156:0/1154788656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 msgr2=0x556ffd40f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 0x556ffd40f1a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2250061130 tx=0x7f225000a000 comp rx=0 tx=0).stop 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 -- 172.21.15.156:0/1154788656 shutdown_connections 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd40f6e0 0x556ffd43f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd43ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 --2- 172.21.15.156:0/1154788656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43c200 0x556ffd40f1a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 -- 172.21.15.156:0/1154788656 >> 172.21.15.156:0/1154788656 conn(0x556ffd411410 msgr2=0x556ffd459f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 -- 172.21.15.156:0/1154788656 shutdown_connections 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.144+0000 7f2267a08740 1 -- 172.21.15.156:0/1154788656 wait complete. 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.145+0000 7f2267a08740 1 -- start start 2024-03-29T04:10:17.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.145+0000 7f2267a08740 1 Processor -- start 2024-03-29T04:10:17.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd42efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd43d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd428f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556ffd413ab0 con 0x556ffd43de20 2024-03-29T04:10:17.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556ffd334cf0 con 0x556ffd43c5e0 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x556ffd4162f0 con 0x556ffd45a800 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f2267a08740 10 client.0 _ll_register_callbacks cb 0x556ffd326f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f225ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd428f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f225ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd428f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42526/0 (socket says 172.21.15.156:42526) 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f225ed9d640 1 -- 172.21.15.156:0/4140340254 learned_addr learned my addr 172.21.15.156:0/4140340254 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.146+0000 7f225e59c640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd42efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225dd9b640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd43d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225ed9d640 1 -- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 msgr2=0x556ffd43d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225ed9d640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd43d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225ed9d640 1 -- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 msgr2=0x556ffd42efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225ed9d640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd42efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225ed9d640 1 -- 172.21.15.156:0/4140340254 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ffd1577d0 con 0x556ffd43de20 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225dd9b640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd43d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:17.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f225ed9d640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd428f20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2258009e50 tx=0x7f2258004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab54ed13fbebc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f223d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f223d7fa640 1 -- 172.21.15.156:0/4140340254 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2258034920 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f225005f070 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f223d7fa640 1 -- 172.21.15.156:0/4140340254 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f2258034ac0 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.147+0000 7f223d7fa640 1 -- 172.21.15.156:0/4140340254 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2258034d20 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 10 client.5577 fetch_fsmap learned FSMap version 17 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x556ffd08d3f0 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f223d7fa640 1 -- 172.21.15.156:0/4140340254 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225005f070 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f223d7fa640 1 -- 172.21.15.156:0/4140340254 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f2258048480 con 0x556ffd43de20 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 10 client.5577 fetch_fsmap finished waiting for FSMap version 17 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 10 client.5577 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 20 client.5577 populate_metadata read hostname 'smithi156' 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd458790 0x556ffd458ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 4 client.5577 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T04:10:17.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.148+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ffd386dc0 con 0x556ffd458790 2024-03-29T04:10:17.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.149+0000 7f225e59c640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd458790 0x556ffd458ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.150+0000 7f225e59c640 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd458790 0x556ffd458ba0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ebebe30de786b21 server_cookie=72a4b3736b71b7ba in_seq=0 out_seq=0 2024-03-29T04:10:17.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.150+0000 7f223d7fa640 10 client.5577 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f223d7fa640 1 -- 172.21.15.156:0/4140340254 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23239 (crc 0 0 0) 0x556ffd386dc0 con 0x556ffd458790 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f223d7fa640 10 client.5577 handle_command_reply: tid=0 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 client.5577 shutdown 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 msgr2=0x556ffd428f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd428f20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2258009e50 tx=0x7f2258004ab0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 shutdown_connections 2024-03-29T04:10:17.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd458790 0x556ffd458ba0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd428f20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd43d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 --2- 172.21.15.156:0/4140340254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd42efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 >> 172.21.15.156:0/4140340254 conn(0x556ffd40f6e0 msgr2=0x556ffd4130e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.153+0000 7f223d7fa640 0 client.5577 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.154+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 shutdown_connections 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.154+0000 7f2267a08740 1 -- 172.21.15.156:0/4140340254 wait complete. 2024-03-29T04:10:17.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.154+0000 7f2267a08740 20 client.5577 trim_cache size 0 max 16384 2024-03-29T04:10:17.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.171+0000 7f2267a08740 1 Processor -- start 2024-03-29T04:10:17.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.171+0000 7f2267a08740 1 -- start start 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd459140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f2267a08740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd456610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd4d5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f2267a08740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2260070430 con 0x556ffd43de20 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22601f53e0 con 0x556ffd43c5e0 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f2260157980 con 0x556ffd45a800 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd456610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd456610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42528/0 (socket says 172.21.15.156:42528) 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 -- 172.21.15.156:0/663738163 learned_addr learned my addr 172.21.15.156:0/663738163 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225e59c640 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd459140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225ed9d640 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd4d5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 -- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 msgr2=0x556ffd4d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd4d5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 -- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 msgr2=0x556ffd459140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd459140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225dd9b640 1 -- 172.21.15.156:0/663738163 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ffd429290 con 0x556ffd43de20 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.172+0000 7f225ed9d640 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd4d5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f225dd9b640 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd456610 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f225400fbf0 tx=0x7f2254012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be1773c2f32344e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f223f7fe640 1 -- 172.21.15.156:0/663738163 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254028070 con 0x556ffd43de20 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f223f7fe640 1 -- 172.21.15.156:0/663738163 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225401b070 con 0x556ffd43de20 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f223f7fe640 1 -- 172.21.15.156:0/663738163 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254023480 con 0x556ffd43de20 2024-03-29T04:10:17.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 -- 172.21.15.156:0/663738163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 msgr2=0x556ffd456610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd456610 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f225400fbf0 tx=0x7f2254012000 comp rx=0 tx=0).stop 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 -- 172.21.15.156:0/663738163 shutdown_connections 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd45a800 0x556ffd4d5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd43de20 0x556ffd456610 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 --2- 172.21.15.156:0/663738163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43c5e0 0x556ffd459140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 -- 172.21.15.156:0/663738163 >> 172.21.15.156:0/663738163 conn(0x556ffd40f6e0 msgr2=0x556ffd432c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.173+0000 7f2267a08740 1 -- 172.21.15.156:0/663738163 shutdown_connections 2024-03-29T04:10:17.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.174+0000 7f2267a08740 1 -- 172.21.15.156:0/663738163 wait complete. 2024-03-29T04:10:17.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.174+0000 7f2267a08740 1 -- start start 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.174+0000 7f2267a08740 1 Processor -- start 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd43c5e0 0x556ffd45d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43de20 0x556ffd4d4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 0x556ffd42c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f22601f4d90 con 0x556ffd45a800 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556ffd42e4f0 con 0x556ffd43de20 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x556ffd47cc50 con 0x556ffd43c5e0 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f2267a08740 10 client.0 _ll_register_callbacks cb 0x7f2258009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 0x556ffd42c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 0x556ffd42c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42542/0 (socket says 172.21.15.156:42542) 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 -- 172.21.15.156:0/1564931193 learned_addr learned my addr 172.21.15.156:0/1564931193 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 -- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd43c5e0 msgr2=0x556ffd45d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd43c5e0 0x556ffd45d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225dd9b640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43de20 0x556ffd4d4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 -- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43de20 msgr2=0x556ffd4d4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43de20 0x556ffd4d4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 -- 172.21.15.156:0/1564931193 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ffd04d0f0 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225dd9b640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43de20 0x556ffd4d4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f225ed9d640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 0x556ffd42c2e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f22580486b0 tx=0x7f2258009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8ea388f2eb13a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.176+0000 7f223d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f223d7fa640 1 -- 172.21.15.156:0/1564931193 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225802cd40 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f223d7fa640 1 -- 172.21.15.156:0/1564931193 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f2258003990 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f225401b070 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f223d7fa640 1 -- 172.21.15.156:0/1564931193 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2258063050 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 10 client.5583 fetch_fsmap learned FSMap version 17 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x556ffd47c990 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f223d7fa640 1 -- 172.21.15.156:0/1564931193 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225401b070 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f223d7fa640 1 -- 172.21.15.156:0/1564931193 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f2258066020 con 0x556ffd45a800 2024-03-29T04:10:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 10 client.5583 fetch_fsmap finished waiting for FSMap version 17 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 10 client.5583 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 20 client.5583 populate_metadata read hostname 'smithi156' 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd47c0b0 0x556ffd4520a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 4 client.5583 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ffd381110 con 0x556ffd47c0b0 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f225e59c640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd47c0b0 0x556ffd4520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.177+0000 7f223d7fa640 10 client.5583 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f225e59c640 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd47c0b0 0x556ffd4520a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0e645c878498eb2 server_cookie=f6296fdfdc3dc563 in_seq=0 out_seq=0 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f223d7fa640 1 -- 172.21.15.156:0/1564931193 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+564 (crc 0 0 0) 0x556ffd381110 con 0x556ffd47c0b0 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f223d7fa640 10 client.5583 handle_command_reply: tid=0 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 client.5583 shutdown 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 msgr2=0x556ffd42c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 0x556ffd42c2e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f22580486b0 tx=0x7f2258009810 comp rx=0 tx=0).stop 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 shutdown_connections 2024-03-29T04:10:17.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x556ffd47c0b0 0x556ffd4520a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ffd45a800 0x556ffd42c2e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ffd43de20 0x556ffd4d4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 --2- 172.21.15.156:0/1564931193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x556ffd43c5e0 0x556ffd45d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 >> 172.21.15.156:0/1564931193 conn(0x556ffd40f6e0 msgr2=0x556ffd458be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f223d7fa640 0 client.5583 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 shutdown_connections 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 1 -- 172.21.15.156:0/1564931193 wait complete. 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.178+0000 7f2267a08740 20 client.5583 trim_cache size 0 max 16384 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:10:17.163 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "scrubs": { 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "d3d9bb08-77ad-4349-b350-47b4ffcee287": { 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "path": "/", 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "tag": "d3d9bb08-77ad-4349-b350-47b4ffcee287", 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "options": "recursive" 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_stats": { 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "d3d9bb08-77ad-4349-b350-47b4ffcee287": { 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "start_time": "2024-03-29T04:10:16.949089+0000", 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "path": "/", 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_started": 0, 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_passed": 0, 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_failed": 0, 2024-03-29T04:10:17.164 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_skipped": 0 2024-03-29T04:10:17.165 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:10:17.165 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:10:17.165 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f224006c530 msgr2=0x7f224006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f224006c530 0x7f224006e9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f22600b4a60 tx=0x7f2254009040 comp rx=0 tx=0).stop 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 msgr2=0x7f22601e8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22601e8d50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f225000c400 tx=0x7f225000c8c0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 shutdown_connections 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f224006c530 0x7f224006e9f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f226015e920 0x7f22601f84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22600d2120 0x7f22601615a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 --2- 172.21.15.156:0/2970071275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22600cfc30 0x7f22601e8d50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 >> 172.21.15.156:0/2970071275 conn(0x7f226009d7a0 msgr2=0x7f22600b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 shutdown_connections 2024-03-29T04:10:17.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:17.179+0000 7f2264e1b640 1 -- 172.21.15.156:0/2970071275 wait complete. 2024-03-29T04:10:17.170 INFO:tasks.cephfs.filesystem:scrub status for tag:d3d9bb08-77ad-4349-b350-47b4ffcee287 - {'path': '/', 'tag': 'd3d9bb08-77ad-4349-b350-47b4ffcee287', 'options': 'recursive'} 2024-03-29T04:10:25.877 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:10:25.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:10:47.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:47.172 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-03-29T04:10:47.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.297+0000 7f40e4cd9640 1 Processor -- start 2024-03-29T04:10:47.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.297+0000 7f40e4cd9640 1 -- start start 2024-03-29T04:10:47.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40e4cd9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40e00d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40e4cd9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e00d2df0 con 0x7f40e00d2120 2024-03-29T04:10:47.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40e4cd9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e00cf190 con 0x7f40e015e920 2024-03-29T04:10:47.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40e4cd9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f40e00cf7a0 con 0x7f40e00cfc30 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40ddd9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f40e00d2120 0x7f40e006c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:38972/0 (socket says 172.21.15.156:38972) 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40ddd9b640 1 -- 172.21.15.156:0/3885116664 learned_addr learned my addr 172.21.15.156:0/3885116664 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40ded9d640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40e00d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4034755101 0 0) 0x7f40e00d2df0 con 0x7f40e00d2120 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c8003520 con 0x7f40e00d2120 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571374498 0 0) 0x7f40e00cf190 con 0x7f40e015e920 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40e00d2df0 con 0x7f40e015e920 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2573089423 0 0) 0x7f40c8003520 con 0x7f40e00d2120 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 msgr2=0x7f40e00d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40e00d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 >> v1:172.21.15.203:6789/0 conn(0x7f40e015e920 legacy=0x7f40e00d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.281 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e00d3080 con 0x7f40e00d2120 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f40d4001670 con 0x7f40e00d2120 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 >> v1:172.21.15.156:6789/0 conn(0x7f40e00d2120 legacy=0x7f40e006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40c8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c8004800 0x7f40c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.298+0000 7f40dd59a640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40e00cf7a0 con 0x7f40c8003b70 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40c800ba70 con 0x7f40c8004800 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f40c800bc40 con 0x7f40e00cfc30 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40de59c640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40c8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40ded9d640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c8004800 0x7f40c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40de59c640 1 -- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 msgr2=0x7f40c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40de59c640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40de59c640 1 -- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c8004800 msgr2=0x7f40c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40de59c640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c8004800 0x7f40c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40de59c640 1 -- 172.21.15.156:0/3885116664 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c8003520 con 0x7f40c8003b70 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c8004800 0x7f40c8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.299+0000 7f40ded9d640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00cfc30 0x7f40c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40de59c640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40c8003f70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40d00018d0 tx=0x7f40c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d505935fee0a24bf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00014f0 con 0x7f40c8003b70 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d002f050 con 0x7f40c8003b70 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40dd59a640 1 -- 172.21.15.156:0/3885116664 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00035f0 con 0x7f40c8003b70 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40e4cd9640 1 -- 172.21.15.156:0/3885116664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 msgr2=0x7f40c8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/3885116664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40c8003f70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40d00018d0 tx=0x7f40c8005830 comp rx=0 tx=0).stop 2024-03-29T04:10:47.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.300+0000 7f40ded9d640 1 -- 172.21.15.156:0/3885116664 reap_dead start 2024-03-29T04:10:47.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.301+0000 7f40e4cd9640 1 -- 172.21.15.156:0/3885116664 shutdown_connections 2024-03-29T04:10:47.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.301+0000 7f40e4cd9640 1 -- 172.21.15.156:0/3885116664 >> 172.21.15.156:0/3885116664 conn(0x7f40e009d7a0 msgr2=0x7f40e00d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:47.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.301+0000 7f40e4cd9640 1 -- 172.21.15.156:0/3885116664 shutdown_connections 2024-03-29T04:10:47.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.301+0000 7f40e4cd9640 1 -- 172.21.15.156:0/3885116664 wait complete. 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.302+0000 7f40e4cd9640 1 Processor -- start 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.302+0000 7f40e4cd9640 1 -- start start 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40e01e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40e01f5e40 0x7f40e01f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00ccbe0 0x7f40e00cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40e006cce0 con 0x7f40c8003b70 2024-03-29T04:10:47.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40e00d3360 con 0x7f40e01f5e40 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f40e00cc750 con 0x7f40e00ccbe0 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40e01e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40e01e8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45534/0 (socket says 172.21.15.156:45534) 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 -- 172.21.15.156:0/2862451428 learned_addr learned my addr 172.21.15.156:0/2862451428 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 -- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00ccbe0 msgr2=0x7f40e00cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40e01f5e40 0x7f40e01f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00ccbe0 0x7f40e00cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 -- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40e01f5e40 msgr2=0x7f40e01f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40e01f5e40 0x7f40e01f62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 -- 172.21.15.156:0/2862451428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e00cbb40 con 0x7f40c8003b70 2024-03-29T04:10:47.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40de59c640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40e01e8280 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f40d00018d0 tx=0x7f40d00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40ceffd640 1 -- 172.21.15.156:0/2862451428 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0043020 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40ceffd640 1 -- 172.21.15.156:0/2862451428 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d0007240 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e00cbe20 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40ceffd640 1 -- 172.21.15.156:0/2862451428 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00014f0 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.303+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e00cc3e0 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.305+0000 7f40ceffd640 1 -- 172.21.15.156:0/2862451428 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f40d0046020 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.305+0000 7f40ceffd640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f40c006c530 0x7f40c006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.305+0000 7f40ceffd640 1 -- 172.21.15.156:0/2862451428 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f40d00ad520 con 0x7f40c8003b70 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.305+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f40c006c530 0x7f40c006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.306+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f40c006c530 0x7f40c006e9f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40d4001eb0 tx=0x7f40d4001130 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.306+0000 7f40e79a6740 1 Processor -- start 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 -- start start 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12bc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab12ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12bc6e0 0x55fab12ec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fab1226880 con 0x55fab12e9200 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fab11ce8f0 con 0x55fab12bc6e0 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x55fab1222830 con 0x55fab12e95e0 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12bc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12bc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45548/0 (socket says 172.21.15.156:45548) 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40de59c640 1 -- 172.21.15.156:0/2545638987 learned_addr learned my addr 172.21.15.156:0/2545638987 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40ded9d640 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12bc6e0 0x55fab12ec4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.307+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab12ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40de59c640 1 -- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 msgr2=0x55fab12ebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40de59c640 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab12ebf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40de59c640 1 -- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12bc6e0 msgr2=0x55fab12ec4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40de59c640 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12bc6e0 0x55fab12ec4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40de59c640 1 -- 172.21.15.156:0/2545638987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fab11d9140 con 0x55fab12e9200 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40de59c640 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12bc1a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f40d0001200 tx=0x7f40d007bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1894afe733caa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40bffff640 1 -- 172.21.15.156:0/2545638987 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0084020 con 0x55fab12e9200 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40bffff640 1 -- 172.21.15.156:0/2545638987 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d002f070 con 0x55fab12e9200 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.308+0000 7f40bffff640 1 -- 172.21.15.156:0/2545638987 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00075f0 con 0x55fab12e9200 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 -- 172.21.15.156:0/2545638987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 msgr2=0x55fab12bc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12bc1a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f40d0001200 tx=0x7f40d007bc40 comp rx=0 tx=0).stop 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 -- 172.21.15.156:0/2545638987 shutdown_connections 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12bc6e0 0x55fab12ec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab12ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 --2- 172.21.15.156:0/2545638987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12bc1a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 -- 172.21.15.156:0/2545638987 >> 172.21.15.156:0/2545638987 conn(0x55fab12be410 msgr2=0x55fab1306f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 -- 172.21.15.156:0/2545638987 shutdown_connections 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 -- 172.21.15.156:0/2545638987 wait complete. 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 -- start start 2024-03-29T04:10:47.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.309+0000 7f40e79a6740 1 Processor -- start 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e95e0 0x55fab12f2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12f1960 0x55fab12f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fab11ce8f0 con 0x55fab12e9200 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fab11e1cf0 con 0x55fab12e95e0 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x55fab1226880 con 0x55fab12f1960 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40e79a6740 10 client.0 _ll_register_callbacks cb 0x55fab11d3f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12ef9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45560/0 (socket says 172.21.15.156:45560) 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40de59c640 1 -- 172.21.15.156:0/1449842984 learned_addr learned my addr 172.21.15.156:0/1449842984 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.310+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e95e0 0x55fab12f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40ded9d640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12f1960 0x55fab12f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40de59c640 1 -- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12f1960 msgr2=0x55fab12f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40de59c640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12f1960 0x55fab12f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40de59c640 1 -- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e95e0 msgr2=0x55fab12f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40de59c640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e95e0 0x55fab12f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40de59c640 1 -- 172.21.15.156:0/1449842984 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fab10047d0 con 0x55fab12e9200 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40de59c640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12ef9f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f40d0007d00 tx=0x7f40d0001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842c27dc4d2b6cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1449842984 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0084020 con 0x55fab12e9200 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1449842984 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f40d00473b0 con 0x55fab12e9200 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40d002f070 con 0x55fab12e9200 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1449842984 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f40d00ad070 con 0x55fab12e9200 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40e79a6740 10 client.5595 fetch_fsmap learned FSMap version 17 2024-03-29T04:10:47.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55fab0f3a3f0 con 0x55fab12e9200 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1449842984 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d002f070 con 0x55fab12e9200 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1449842984 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f40d007a890 con 0x55fab12e9200 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.311+0000 7f40e79a6740 10 client.5595 fetch_fsmap finished waiting for FSMap version 17 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40e79a6740 10 client.5595 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40e79a6740 20 client.5595 populate_metadata read hostname 'smithi156' 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12ed130 0x55fab12ed530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40e79a6740 4 client.5595 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fab1233dc0 con 0x55fab12ed130 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12ed130 0x55fab12ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40bd7fa640 10 client.5595 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.312+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12ed130 0x55fab12ed530 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cd8a06d50b71974 server_cookie=875cdb19a4f7b843 in_seq=0 out_seq=0 2024-03-29T04:10:47.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1449842984 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23239 (crc 0 0 0) 0x55fab1233dc0 con 0x55fab12ed130 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40bd7fa640 10 client.5595 handle_command_reply: tid=0 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 client.5595 shutdown 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 msgr2=0x55fab12ef9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12ef9f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f40d0007d00 tx=0x7f40d0001150 comp rx=0 tx=0).stop 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 shutdown_connections 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12ed130 0x55fab12ed530 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12f1960 0x55fab12f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40bd7fa640 0 client.5595 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e95e0 0x55fab12f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1449842984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12ef9f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 >> 172.21.15.156:0/1449842984 conn(0x55fab12bc6e0 msgr2=0x55fab12c2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 shutdown_connections 2024-03-29T04:10:47.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 1 -- 172.21.15.156:0/1449842984 wait complete. 2024-03-29T04:10:47.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.313+0000 7f40e79a6740 20 client.5595 trim_cache size 0 max 16384 2024-03-29T04:10:47.310 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 Processor -- start 2024-03-29T04:10:47.310 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 -- start start 2024-03-29T04:10:47.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12e61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab13198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12ed130 0x55fab12bb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.312 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40e02013b0 con 0x55fab12e9200 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01f9090 con 0x55fab12ed130 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f40e01f83f0 con 0x55fab12e95e0 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12e61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40de59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12e61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45568/0 (socket says 172.21.15.156:45568) 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.331+0000 7f40de59c640 1 -- 172.21.15.156:0/1601743338 learned_addr learned my addr 172.21.15.156:0/1601743338 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40ded9d640 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12ed130 0x55fab12bb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40de59c640 1 -- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 msgr2=0x55fab13198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40de59c640 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab13198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40de59c640 1 -- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12ed130 msgr2=0x55fab12bb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40de59c640 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12ed130 0x55fab12bb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40de59c640 1 -- 172.21.15.156:0/1601743338 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fab0efa0f0 con 0x55fab12e9200 2024-03-29T04:10:47.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40de59c640 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12e61e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f40d0083af0 tx=0x7f40d0035000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ade175a4a272168 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40bf7fe640 1 -- 172.21.15.156:0/1601743338 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00ab990 con 0x55fab12e9200 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40bf7fe640 1 -- 172.21.15.156:0/1601743338 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d007bb40 con 0x55fab12e9200 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40bf7fe640 1 -- 172.21.15.156:0/1601743338 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0084020 con 0x55fab12e9200 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40e79a6740 1 -- 172.21.15.156:0/1601743338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 msgr2=0x55fab12e61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12e61e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f40d0083af0 tx=0x7f40d0035000 comp rx=0 tx=0).stop 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40e79a6740 1 -- 172.21.15.156:0/1601743338 shutdown_connections 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.332+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12ed130 0x55fab12bb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab13198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1601743338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12e9200 0x55fab12e61e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 -- 172.21.15.156:0/1601743338 >> 172.21.15.156:0/1601743338 conn(0x55fab12bc6e0 msgr2=0x55fab12e5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 -- 172.21.15.156:0/1601743338 shutdown_connections 2024-03-29T04:10:47.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 -- 172.21.15.156:0/1601743338 wait complete. 2024-03-29T04:10:47.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 -- start start 2024-03-29T04:10:47.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.333+0000 7f40e79a6740 1 Processor -- start 2024-03-29T04:10:47.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e9200 0x55fab130a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab131a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 0x55fab131a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fab132b230 con 0x55fab12ed130 2024-03-29T04:10:47.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01f8a40 con 0x55fab12e9200 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x55fab132aed0 con 0x55fab12e95e0 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40e79a6740 10 client.0 _ll_register_callbacks cb 0x7f40d8009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 0x55fab131a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 0x55fab131a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45576/0 (socket says 172.21.15.156:45576) 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 -- 172.21.15.156:0/1442205927 learned_addr learned my addr 172.21.15.156:0/1442205927 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40de59c640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e9200 0x55fab130a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ddd9b640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab131a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 -- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 msgr2=0x55fab131a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab131a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 -- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e9200 msgr2=0x55fab130a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e9200 0x55fab130a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 -- 172.21.15.156:0/1442205927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fab12e7a30 con 0x55fab12ed130 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40de59c640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e9200 0x55fab130a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:47.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40ded9d640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 0x55fab131a990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f40e008c860 tx=0x7f40d8015700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d261638aa636018 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.335+0000 7f40bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1442205927 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d801e070 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1442205927 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f40d8018620 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40d007bb40 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1442205927 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f40d8018820 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 10 client.5601 fetch_fsmap learned FSMap version 17 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55fab1382700 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1442205927 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d007bb40 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1442205927 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f40d802f080 con 0x55fab12ed130 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 10 client.5601 fetch_fsmap finished waiting for FSMap version 17 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 10 client.5601 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 20 client.5601 populate_metadata read hostname 'smithi156' 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12f1960 0x55fab12f2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:47.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 4 client.5601 mds_command: new command op to 5310 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 --> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fab12f4600 con 0x55fab12f1960 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40de59c640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12f1960 0x55fab12f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40bd7fa640 10 client.5601 ms_handle_connect on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.336+0000 7f40de59c640 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12f1960 0x55fab12f2400 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99499d1b7aaee53a server_cookie=8f01976726f4a93 in_seq=0 out_seq=0 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40bd7fa640 1 -- 172.21.15.156:0/1442205927 <== mds.0 v2:172.21.15.156:6835/788767406 1 ==== command_reply(tid 0: 0 ) ==== 8+0+376 (crc 0 0 0) 0x55fab12f4600 con 0x55fab12f1960 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40bd7fa640 10 client.5601 handle_command_reply: tid=0 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 client.5601 shutdown 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 msgr2=0x55fab131a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 0x55fab131a990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f40e008c860 tx=0x7f40d8015700 comp rx=0 tx=0).stop 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 shutdown_connections 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:6835/788767406,v1:172.21.15.156:6837/788767406] conn(0x55fab12f1960 0x55fab12f2400 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fab12ed130 0x55fab131a990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55fab12e95e0 0x55fab131a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 --2- 172.21.15.156:0/1442205927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fab12e9200 0x55fab130a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40bd7fa640 0 client.5601 ms_handle_reset on v2:172.21.15.156:6835/788767406 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 >> 172.21.15.156:0/1442205927 conn(0x55fab12bc6e0 msgr2=0x55fab12e5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 shutdown_connections 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 1 -- 172.21.15.156:0/1442205927 wait complete. 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e79a6740 20 client.5601 trim_cache size 0 max 16384 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "no active scrubs running", 2024-03-29T04:10:47.323 INFO:teuthology.orchestra.run.smithi156.stdout: "scrubs": {}, 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_stats": { 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "d3d9bb08-77ad-4349-b350-47b4ffcee287": { 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "start_time": "2024-03-29T04:10:16.949089+0000", 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "path": "/", 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_started": 23, 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_passed": 23, 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_failed": 0, 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: "uninline_skipped": 0 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-29T04:10:47.324 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:10:47.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f40c006c530 msgr2=0x7f40c006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f40c006c530 0x7f40c006e9f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40d4001eb0 tx=0x7f40d4001130 comp rx=0 tx=0).stop 2024-03-29T04:10:47.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 msgr2=0x7f40e01e8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:47.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.337+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40e01e8280 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f40d00018d0 tx=0x7f40d00012d0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 shutdown_connections 2024-03-29T04:10:47.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f40c006c530 0x7f40c006e9f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f40e00ccbe0 0x7f40e00cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40e01f5e40 0x7f40e01f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 --2- 172.21.15.156:0/2862451428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40c8003b70 0x7f40e01e8280 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:47.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 >> 172.21.15.156:0/2862451428 conn(0x7f40e009d7a0 msgr2=0x7f40e00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:47.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 shutdown_connections 2024-03-29T04:10:47.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:47.338+0000 7f40e4cd9640 1 -- 172.21.15.156:0/2862451428 wait complete. 2024-03-29T04:10:47.328 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-29T04:10:47.328 DEBUG:tasks.cephfs.test_uninlining:extracting inodes 2024-03-29T04:10:47.328 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ceph/Kconfig 2024-03-29T04:10:47.328 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ceph/Kconfig 2024-03-29T04:10:47.345 INFO:teuthology.orchestra.run.smithi156.stdout:1099511627780 linux-5.4/fs/ceph/Kconfig 2024-03-29T04:10:47.346 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ceph/Makefile 2024-03-29T04:10:47.346 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ceph/Makefile 2024-03-29T04:10:47.403 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628281 linux-5.4/fs/ceph/Makefile 2024-03-29T04:10:47.403 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ceph/ceph_frag.c 2024-03-29T04:10:47.403 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ceph/ceph_frag.c 2024-03-29T04:10:47.460 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628287 linux-5.4/fs/ceph/ceph_frag.c 2024-03-29T04:10:47.460 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ceph/io.h 2024-03-29T04:10:47.460 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ceph/io.h 2024-03-29T04:10:47.516 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628294 linux-5.4/fs/ceph/io.h 2024-03-29T04:10:47.516 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ceph/ioctl.h 2024-03-29T04:10:47.516 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ceph/ioctl.h 2024-03-29T04:10:47.573 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628296 linux-5.4/fs/ceph/ioctl.h 2024-03-29T04:10:47.573 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/Kconfig 2024-03-29T04:10:47.573 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/Kconfig 2024-03-29T04:10:47.629 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628308 linux-5.4/fs/ext2/Kconfig 2024-03-29T04:10:47.629 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/xattr_user.c 2024-03-29T04:10:47.630 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/xattr_user.c 2024-03-29T04:10:47.686 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628326 linux-5.4/fs/ext2/xattr_user.c 2024-03-29T04:10:47.687 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/xattr_security.c 2024-03-29T04:10:47.687 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/xattr_security.c 2024-03-29T04:10:47.744 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628324 linux-5.4/fs/ext2/xattr_security.c 2024-03-29T04:10:47.744 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/Makefile 2024-03-29T04:10:47.744 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/Makefile 2024-03-29T04:10:47.801 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628309 linux-5.4/fs/ext2/Makefile 2024-03-29T04:10:47.801 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/symlink.c 2024-03-29T04:10:47.801 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/symlink.c 2024-03-29T04:10:47.857 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628321 linux-5.4/fs/ext2/symlink.c 2024-03-29T04:10:47.857 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/acl.h 2024-03-29T04:10:47.857 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/acl.h 2024-03-29T04:10:47.913 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628311 linux-5.4/fs/ext2/acl.h 2024-03-29T04:10:47.913 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/xattr_trusted.c 2024-03-29T04:10:47.913 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/xattr_trusted.c 2024-03-29T04:10:47.969 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628325 linux-5.4/fs/ext2/xattr_trusted.c 2024-03-29T04:10:47.969 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/ext2/xattr.h 2024-03-29T04:10:47.970 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/ext2/xattr.h 2024-03-29T04:10:48.026 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628323 linux-5.4/fs/ext2/xattr.h 2024-03-29T04:10:48.026 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/Kconfig 2024-03-29T04:10:48.026 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/Kconfig 2024-03-29T04:10:48.082 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628328 linux-5.4/fs/orangefs/Kconfig 2024-03-29T04:10:48.082 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/orangefs-debug.h 2024-03-29T04:10:48.083 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/orangefs-debug.h 2024-03-29T04:10:48.142 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628341 linux-5.4/fs/orangefs/orangefs-debug.h 2024-03-29T04:10:48.142 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/orangefs-sysfs.h 2024-03-29T04:10:48.142 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/orangefs-sysfs.h 2024-03-29T04:10:48.198 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628348 linux-5.4/fs/orangefs/orangefs-sysfs.h 2024-03-29T04:10:48.199 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/dcache.c 2024-03-29T04:10:48.199 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/dcache.c 2024-03-29T04:10:48.255 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628331 linux-5.4/fs/orangefs/dcache.c 2024-03-29T04:10:48.255 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/orangefs-bufmap.h 2024-03-29T04:10:48.255 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/orangefs-bufmap.h 2024-03-29T04:10:48.312 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628339 linux-5.4/fs/orangefs/orangefs-bufmap.h 2024-03-29T04:10:48.312 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/orangefs-debugfs.h 2024-03-29T04:10:48.312 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/orangefs-debugfs.h 2024-03-29T04:10:48.369 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628343 linux-5.4/fs/orangefs/orangefs-debugfs.h 2024-03-29T04:10:48.369 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/orangefs-dev-proto.h 2024-03-29T04:10:48.370 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/orangefs-dev-proto.h 2024-03-29T04:10:48.426 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628344 linux-5.4/fs/orangefs/orangefs-dev-proto.h 2024-03-29T04:10:48.426 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/Makefile 2024-03-29T04:10:48.426 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/Makefile 2024-03-29T04:10:48.484 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628329 linux-5.4/fs/orangefs/Makefile 2024-03-29T04:10:48.484 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/symlink.c 2024-03-29T04:10:48.484 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/symlink.c 2024-03-29T04:10:48.540 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628352 linux-5.4/fs/orangefs/symlink.c 2024-03-29T04:10:48.540 DEBUG:tasks.cephfs.test_uninlining:getting inode for:linux-5.4/fs/orangefs/downcall.h 2024-03-29T04:10:48.540 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.0 && ls -i linux-5.4/fs/orangefs/downcall.h 2024-03-29T04:10:48.596 INFO:teuthology.orchestra.run.smithi156.stdout:1099511628334 linux-5.4/fs/orangefs/downcall.h 2024-03-29T04:10:48.597 DEBUG:tasks.cephfs.test_uninlining:getting inline data info 2024-03-29T04:10:48.597 DEBUG:tasks.cephfs.test_uninlining:getting inode info #1 of 23:1099511627780 2024-03-29T04:10:48.597 DEBUG:tasks.cephfs.test_uninlining:try #1 - dump inode 1099511627780 2024-03-29T04:10:48.597 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T04:10:48.597 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T04:10:48.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T04:10:48.730 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T04:10:48.733 INFO:tasks.cephfs.mount:Terminating background process 2024-03-29T04:10:48.733 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:10:48.733 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:10:48.748 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:10:48.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.911+0000 7f4a3771c640 1 Processor -- start 2024-03-29T04:10:48.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.911+0000 7f4a3771c640 1 -- start start 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a3771c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4a3006dbc0 0x7f4a30169350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a3771c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a300cfa20 con 0x7f4a3015e730 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a3771c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a300cfd00 con 0x7f4a300d42c0 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a3771c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f4a3006c720 con 0x7f4a3006dbc0 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a35cb9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4a3006dbc0 0x7f4a30169350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a354b8640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4a3015e730 0x7f4a300ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:38978/0 (socket says 172.21.15.156:38978) 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.912+0000 7f4a354b8640 1 -- 172.21.15.156:0/2551159067 learned_addr learned my addr 172.21.15.156:0/2551159067 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a35cb9640 1 -- 172.21.15.156:0/2551159067 >> v1:172.21.15.203:6789/0 conn(0x7f4a300d42c0 legacy=0x7f4a300d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a35cb9640 1 -- 172.21.15.156:0/2551159067 >> v1:172.21.15.156:6789/0 conn(0x7f4a3015e730 legacy=0x7f4a300ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a35cb9640 1 -- 172.21.15.156:0/2551159067 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a300cffe0 con 0x7f4a3006dbc0 2024-03-29T04:10:48.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a27fff640 1 -- 172.21.15.156:0/2551159067 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336676940 0 0) 0x7f4a300cfa20 con 0x7f4a3015e730 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a35cb9640 1 --2- 172.21.15.156:0/2551159067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a3006dbc0 0x7f4a30169350 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009d90 tx=0x7f4a2802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10731757a636d851 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a27fff640 1 -- 172.21.15.156:0/2551159067 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28039070 con 0x7f4a3006dbc0 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a27fff640 1 -- 172.21.15.156:0/2551159067 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a2802cd00 con 0x7f4a3006dbc0 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.913+0000 7f4a27fff640 1 -- 172.21.15.156:0/2551159067 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28034660 con 0x7f4a3006dbc0 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 -- 172.21.15.156:0/2551159067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a3006dbc0 msgr2=0x7f4a30169350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 --2- 172.21.15.156:0/2551159067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a3006dbc0 0x7f4a30169350 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009d90 tx=0x7f4a2802bf50 comp rx=0 tx=0).stop 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 -- 172.21.15.156:0/2551159067 shutdown_connections 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 --2- 172.21.15.156:0/2551159067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a3006dbc0 0x7f4a30169350 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 -- 172.21.15.156:0/2551159067 >> 172.21.15.156:0/2551159067 conn(0x7f4a3008bfd0 msgr2=0x7f4a3009d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 -- 172.21.15.156:0/2551159067 shutdown_connections 2024-03-29T04:10:48.894 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.914+0000 7f4a3771c640 1 -- 172.21.15.156:0/2551159067 wait complete. 2024-03-29T04:10:48.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.916+0000 7f4a3771c640 1 Processor -- start 2024-03-29T04:10:48.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.916+0000 7f4a3771c640 1 -- start start 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a3771c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a3006dbc0 0x7f4a30153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a3771c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 0x7f4a300cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a3771c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4a3015e730 0x7f4a300cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a3771c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3015eb10 con 0x7f4a300d42c0 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a3771c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a300d2220 con 0x7f4a3006dbc0 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a3771c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3015ec90 con 0x7f4a3015e730 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a34cb7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 0x7f4a300cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a34cb7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 0x7f4a300cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45600/0 (socket says 172.21.15.156:45600) 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a34cb7640 1 -- 172.21.15.156:0/1078204525 learned_addr learned my addr 172.21.15.156:0/1078204525 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.917+0000 7f4a354b8640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a3006dbc0 0x7f4a30153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a35cb9640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4a3015e730 0x7f4a300cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a34cb7640 1 -- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4a3015e730 msgr2=0x7f4a300cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:48.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a34cb7640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4a3015e730 0x7f4a300cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:48.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a34cb7640 1 -- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a3006dbc0 msgr2=0x7f4a30153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:48.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a34cb7640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a3006dbc0 0x7f4a30153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:48.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a34cb7640 1 -- 172.21.15.156:0/1078204525 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a300c9230 con 0x7f4a300d42c0 2024-03-29T04:10:48.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a34cb7640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 0x7f4a300cbd80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c450 tx=0x7f4a2c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:48.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c0052d0 con 0x7f4a300d42c0 2024-03-29T04:10:48.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a2c005490 con 0x7f4a300d42c0 2024-03-29T04:10:48.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c003880 con 0x7f4a300d42c0 2024-03-29T04:10:48.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a30161310 con 0x7f4a300d42c0 2024-03-29T04:10:48.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.918+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a301618d0 con 0x7f4a300d42c0 2024-03-29T04:10:48.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.919+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a30068610 con 0x7f4a300d42c0 2024-03-29T04:10:48.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.919+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4a2c003a20 con 0x7f4a300d42c0 2024-03-29T04:10:48.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.920+0000 7f4a25ffb640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4a1406c530 0x7f4a1406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:48.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.920+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4a2c08b2d0 con 0x7f4a300d42c0 2024-03-29T04:10:48.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.920+0000 7f4a354b8640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4a1406c530 0x7f4a1406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:48.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.920+0000 7f4a354b8640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4a1406c530 0x7f4a1406e9f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4a20000f60 tx=0x7f4a20007d00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:48.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:48.924+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4a2c059980 con 0x7f4a300d42c0 2024-03-29T04:10:49.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.146+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a30161050 con 0x7f4a300d42c0 2024-03-29T04:10:49.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.147+0000 7f4a25ffb640 1 -- 172.21.15.156:0/1078204525 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7f4a2c059280 con 0x7f4a300d42c0 2024-03-29T04:10:49.126 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:49.126 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":34,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:55.015332+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:10:49.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4a1406c530 msgr2=0x7f4a1406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4a1406c530 0x7f4a1406e9f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4a20000f60 tx=0x7f4a20007d00 comp rx=0 tx=0).stop 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 msgr2=0x7f4a300cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 0x7f4a300cbd80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c450 tx=0x7f4a2c00c910 comp rx=0 tx=0).stop 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 shutdown_connections 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4a1406c530 0x7f4a1406e9f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4a3015e730 0x7f4a300cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a300d42c0 0x7f4a300cbd80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 --2- 172.21.15.156:0/1078204525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a3006dbc0 0x7f4a30153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 >> 172.21.15.156:0/1078204525 conn(0x7f4a3008bfd0 msgr2=0x7f4a300b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 shutdown_connections 2024-03-29T04:10:49.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.149+0000 7f4a3771c640 1 -- 172.21.15.156:0/1078204525 wait complete. 2024-03-29T04:10:49.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:09:51.822722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:09:52.315661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:10:49.143 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:10:49.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 Processor -- start 2024-03-29T04:10:49.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 -- start start 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c15e920 0x7eff7c0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff7c0d2120 0x7eff7c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff7c06cb10 0x7eff7c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7eff7c0cf7a0 con 0x7eff7c0d2120 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7eff7c0cf190 con 0x7eff7c06cb10 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.271+0000 7eff817d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7eff7c0d3080 con 0x7eff7c15e920 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff7c0d2120 0x7eff7c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff7c0d2120 0x7eff7c06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45608/0 (socket says 172.21.15.156:45608) 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 -- 172.21.15.156:0/2426247681 learned_addr learned my addr 172.21.15.156:0/2426247681 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff7b7fe640 1 --2- 172.21.15.156:0/2426247681 >> v2:172.21.15.203:3300/0 conn(0x7eff7c06cb10 0x7eff7c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff7affd640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c15e920 0x7eff7c0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 -- 172.21.15.156:0/2426247681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c15e920 msgr2=0x7eff7c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c15e920 0x7eff7c0d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 -- 172.21.15.156:0/2426247681 >> v2:172.21.15.203:3300/0 conn(0x7eff7c06cb10 msgr2=0x7eff7c0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 --2- 172.21.15.156:0/2426247681 >> v2:172.21.15.203:3300/0 conn(0x7eff7c06cb10 0x7eff7c0cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff73fff640 1 -- 172.21.15.156:0/2426247681 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c0cf430 con 0x7eff7c0d2120 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.272+0000 7eff7affd640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c15e920 0x7eff7c0d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff73fff640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c0d2120 0x7eff7c06c5d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0b5ed0 tx=0x7eff6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937448152081f7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff7a7fc640 1 -- 172.21.15.156:0/2426247681 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64035020 con 0x7eff7c0d2120 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff7a7fc640 1 -- 172.21.15.156:0/2426247681 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff64031870 con 0x7eff7c0d2120 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff7a7fc640 1 -- 172.21.15.156:0/2426247681 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64031c40 con 0x7eff7c0d2120 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 -- 172.21.15.156:0/2426247681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c0d2120 msgr2=0x7eff7c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c0d2120 0x7eff7c06c5d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0b5ed0 tx=0x7eff6402b910 comp rx=0 tx=0).stop 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 -- 172.21.15.156:0/2426247681 shutdown_connections 2024-03-29T04:10:49.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c0d2120 0x7eff7c06c5d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 --2- 172.21.15.156:0/2426247681 >> v2:172.21.15.203:3300/0 conn(0x7eff7c06cb10 0x7eff7c0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 --2- 172.21.15.156:0/2426247681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c15e920 0x7eff7c0d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.273+0000 7eff817d4640 1 -- 172.21.15.156:0/2426247681 >> 172.21.15.156:0/2426247681 conn(0x7eff7c09d7a0 msgr2=0x7eff7c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:49.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.274+0000 7eff817d4640 1 -- 172.21.15.156:0/2426247681 shutdown_connections 2024-03-29T04:10:49.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.274+0000 7eff817d4640 1 -- 172.21.15.156:0/2426247681 wait complete. 2024-03-29T04:10:49.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.275+0000 7eff817d4640 1 Processor -- start 2024-03-29T04:10:49.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.275+0000 7eff817d4640 1 -- start start 2024-03-29T04:10:49.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff7c06cb10 0x7eff7c0b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c0d2120 0x7eff7c0b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 0x7eff7c1f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c0d2da0 con 0x7eff7c15e920 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c0d2c20 con 0x7eff7c06cb10 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7eff7c0d0170 con 0x7eff7c0d2120 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 0x7eff7c1f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 0x7eff7c1f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45620/0 (socket says 172.21.15.156:45620) 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 -- 172.21.15.156:0/463312203 learned_addr learned my addr 172.21.15.156:0/463312203 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7affd640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff7c06cb10 0x7eff7c0b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff73fff640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c0d2120 0x7eff7c0b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 -- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c0d2120 msgr2=0x7eff7c0b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c0d2120 0x7eff7c0b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 -- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff7c06cb10 msgr2=0x7eff7c0b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff7c06cb10 0x7eff7c0b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 -- 172.21.15.156:0/463312203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c0b7110 con 0x7eff7c15e920 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff7b7fe640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 0x7eff7c1f4320 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7eff74007e30 tx=0x7eff7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74014420 con 0x7eff7c15e920 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff74019070 con 0x7eff7c15e920 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c0b73f0 con 0x7eff7c15e920 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74014800 con 0x7eff7c15e920 2024-03-29T04:10:49.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.277+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c0b79b0 con 0x7eff7c15e920 2024-03-29T04:10:49.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.279+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff7c0a2480 con 0x7eff7c15e920 2024-03-29T04:10:49.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.279+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7eff740149a0 con 0x7eff7c15e920 2024-03-29T04:10:49.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.279+0000 7eff737fe640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff5c06c530 0x7eff5c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.279+0000 7eff7affd640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff5c06c530 0x7eff5c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.279+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7eff7408bf40 con 0x7eff7c15e920 2024-03-29T04:10:49.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.280+0000 7eff7affd640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff5c06c530 0x7eff5c06e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00a950 tx=0x7eff6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.284+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7eff7405a560 con 0x7eff7c15e920 2024-03-29T04:10:49.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.461+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff7c0d1d80 con 0x7eff7c15e920 2024-03-29T04:10:49.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.461+0000 7eff737fe640 1 -- 172.21.15.156:0/463312203 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7eff74059e60 con 0x7eff7c15e920 2024-03-29T04:10:49.440 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:10:49.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff5c06c530 msgr2=0x7eff5c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff5c06c530 0x7eff5c06e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00a950 tx=0x7eff6c008040 comp rx=0 tx=0).stop 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 msgr2=0x7eff7c1f4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 0x7eff7c1f4320 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7eff74007e30 tx=0x7eff7400e4b0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 shutdown_connections 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7eff5c06c530 0x7eff5c06e9f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff7c15e920 0x7eff7c1f4320 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff7c0d2120 0x7eff7c0b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 --2- 172.21.15.156:0/463312203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff7c06cb10 0x7eff7c0b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 >> 172.21.15.156:0/463312203 conn(0x7eff7c09d7a0 msgr2=0x7eff7c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 shutdown_connections 2024-03-29T04:10:49.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.463+0000 7eff817d4640 1 -- 172.21.15.156:0/463312203 wait complete. 2024-03-29T04:10:49.453 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:10:49.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.579+0000 7f44218c8640 1 Processor -- start 2024-03-29T04:10:49.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.579+0000 7f44218c8640 1 -- start start 2024-03-29T04:10:49.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f44218c8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c15fe40 0x7f441c0d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f44218c8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f441c0d01f0 0x7f441c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f44218c8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f441c09aa40 0x7f441c09ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f44218c8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f441c0d1470 con 0x7f441c09aa40 2024-03-29T04:10:49.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f44218c8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f441c0b3c40 con 0x7f441c0d01f0 2024-03-29T04:10:49.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f44218c8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f441c09b3d0 con 0x7f441c15fe40 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f441c09aa40 0x7f441c09ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f441c09aa40 0x7f441c09ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45622/0 (socket says 172.21.15.156:45622) 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 -- 172.21.15.156:0/3500286503 learned_addr learned my addr 172.21.15.156:0/3500286503 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441a7fc640 1 --2- 172.21.15.156:0/3500286503 >> v2:172.21.15.203:3300/0 conn(0x7f441c0d01f0 0x7f441c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441affd640 1 --2- 172.21.15.156:0/3500286503 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c15fe40 0x7f441c0d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 -- 172.21.15.156:0/3500286503 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c15fe40 msgr2=0x7f441c0d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 --2- 172.21.15.156:0/3500286503 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c15fe40 0x7f441c0d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 -- 172.21.15.156:0/3500286503 >> v2:172.21.15.203:3300/0 conn(0x7f441c0d01f0 msgr2=0x7f441c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 --2- 172.21.15.156:0/3500286503 >> v2:172.21.15.203:3300/0 conn(0x7f441c0d01f0 0x7f441c0b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441b7fe640 1 -- 172.21.15.156:0/3500286503 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441c0b17e0 con 0x7f441c09aa40 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.580+0000 7f441a7fc640 1 --2- 172.21.15.156:0/3500286503 >> v2:172.21.15.203:3300/0 conn(0x7f441c0d01f0 0x7f441c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.581+0000 7f441b7fe640 1 --2- 172.21.15.156:0/3500286503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c09aa40 0x7f441c09ae90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f441c073080 tx=0x7f440c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81c1269958c1909 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.581+0000 7f4419ffb640 1 -- 172.21.15.156:0/3500286503 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c02c950 con 0x7f441c09aa40 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.581+0000 7f4419ffb640 1 -- 172.21.15.156:0/3500286503 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440c02caf0 con 0x7f441c09aa40 2024-03-29T04:10:49.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.581+0000 7f4419ffb640 1 -- 172.21.15.156:0/3500286503 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c0343f0 con 0x7f441c09aa40 2024-03-29T04:10:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 -- 172.21.15.156:0/3500286503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c09aa40 msgr2=0x7f441c09ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 --2- 172.21.15.156:0/3500286503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c09aa40 0x7f441c09ae90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f441c073080 tx=0x7f440c02bdb0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 -- 172.21.15.156:0/3500286503 shutdown_connections 2024-03-29T04:10:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 --2- 172.21.15.156:0/3500286503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c09aa40 0x7f441c09ae90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 --2- 172.21.15.156:0/3500286503 >> v2:172.21.15.203:3300/0 conn(0x7f441c0d01f0 0x7f441c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 --2- 172.21.15.156:0/3500286503 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c15fe40 0x7f441c0d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 -- 172.21.15.156:0/3500286503 >> 172.21.15.156:0/3500286503 conn(0x7f441c0b3de0 msgr2=0x7f441c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 -- 172.21.15.156:0/3500286503 shutdown_connections 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.582+0000 7f44218c8640 1 -- 172.21.15.156:0/3500286503 wait complete. 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.583+0000 7f44218c8640 1 Processor -- start 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.583+0000 7f44218c8640 1 -- start start 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f44218c8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f441c09aa40 0x7f441c1507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f44218c8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c0d01f0 0x7f441c0b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f44218c8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 0x7f441c0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f44218c8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f441c0ce190 con 0x7f441c15fe40 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f44218c8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f441c0ce750 con 0x7f441c09aa40 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f44218c8640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f441c0ce470 con 0x7f441c0d01f0 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 0x7f441c0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 0x7f441c0b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45634/0 (socket says 172.21.15.156:45634) 2024-03-29T04:10:49.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 -- 172.21.15.156:0/1060798859 learned_addr learned my addr 172.21.15.156:0/1060798859 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441affd640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f441c09aa40 0x7f441c1507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441a7fc640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c0d01f0 0x7f441c0b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 -- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c0d01f0 msgr2=0x7f441c0b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c0d01f0 0x7f441c0b7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 -- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f441c09aa40 msgr2=0x7f441c1507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f441c09aa40 0x7f441c1507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 -- 172.21.15.156:0/1060798859 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441c06da70 con 0x7f441c15fe40 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441affd640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f441c09aa40 0x7f441c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.584+0000 7f441b7fe640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 0x7f441c0b5140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f440c036040 tx=0x7f440c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.585+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c043020 con 0x7f441c15fe40 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.585+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440c03e450 con 0x7f441c15fe40 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.585+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441c06dd50 con 0x7f441c15fe40 2024-03-29T04:10:49.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.585+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c009510 con 0x7f441c15fe40 2024-03-29T04:10:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.585+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f441c0b0730 con 0x7f441c15fe40 2024-03-29T04:10:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.586+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f441c0a03c0 con 0x7f441c15fe40 2024-03-29T04:10:49.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.586+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f440c046020 con 0x7f441c15fe40 2024-03-29T04:10:49.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.587+0000 7f4403fff640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f440806c530 0x7f440806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.587+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f440c0ae220 con 0x7f441c15fe40 2024-03-29T04:10:49.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.587+0000 7f441affd640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f440806c530 0x7f440806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.587+0000 7f441affd640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f440806c530 0x7f440806e9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4414006fd0 tx=0x7f4414008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.591+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f440c04d030 con 0x7f441c15fe40 2024-03-29T04:10:49.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.773+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f441c04f9d0 con 0x7f441c15fe40 2024-03-29T04:10:49.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.774+0000 7f4403fff640 1 -- 172.21.15.156:0/1060798859 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f441c0b17e0 con 0x7f441c15fe40 2024-03-29T04:10:49.754 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:10:49.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f440806c530 msgr2=0x7f440806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f440806c530 0x7f440806e9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4414006fd0 tx=0x7f4414008040 comp rx=0 tx=0).stop 2024-03-29T04:10:49.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 msgr2=0x7f441c0b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 0x7f441c0b5140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f440c036040 tx=0x7f440c004a40 comp rx=0 tx=0).stop 2024-03-29T04:10:49.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 shutdown_connections 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f440806c530 0x7f440806e9f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f441c15fe40 0x7f441c0b5140 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f441c0d01f0 0x7f441c0b7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 --2- 172.21.15.156:0/1060798859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f441c09aa40 0x7f441c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 >> 172.21.15.156:0/1060798859 conn(0x7f441c0b3de0 msgr2=0x7f441c0b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 shutdown_connections 2024-03-29T04:10:49.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.776+0000 7f44218c8640 1 -- 172.21.15.156:0/1060798859 wait complete. 2024-03-29T04:10:49.767 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:10:49.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.891+0000 7fead07f0640 1 Processor -- start 2024-03-29T04:10:49.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.891+0000 7fead07f0640 1 -- start start 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7fead07f0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7feac81415d0 0x7feac815eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7fead07f0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7feac80b3dd0 0x7feac80b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7fead07f0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac8162130 con 0x7feac8144650 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7fead07f0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7feac80b4720 con 0x7feac80b3dd0 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7fead07f0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7feac81422c0 con 0x7feac81415d0 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feaced8d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7feac80b3dd0 0x7feac80b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feacdd8b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7feac8144650 0x7feac8141bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:38988/0 (socket says 172.21.15.156:38988) 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feacdd8b640 1 -- 172.21.15.156:0/1286660021 learned_addr learned my addr 172.21.15.156:0/1286660021 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feace58c640 1 --2- 172.21.15.156:0/1286660021 >> v2:172.21.15.203:3300/0 conn(0x7feac81415d0 0x7feac815eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feaced8d640 1 -- 172.21.15.156:0/1286660021 >> v2:172.21.15.203:3300/0 conn(0x7feac81415d0 msgr2=0x7feac815eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feaced8d640 1 --2- 172.21.15.156:0/1286660021 >> v2:172.21.15.203:3300/0 conn(0x7feac81415d0 0x7feac815eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feaced8d640 1 -- 172.21.15.156:0/1286660021 >> v1:172.21.15.156:6789/0 conn(0x7feac8144650 legacy=0x7feac8141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feaced8d640 1 -- 172.21.15.156:0/1286660021 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac815f0f0 con 0x7feac80b3dd0 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feacd58a640 1 -- 172.21.15.156:0/1286660021 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423492366 0 0) 0x7feac8162130 con 0x7feac8144650 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.892+0000 7feace58c640 1 --2- 172.21.15.156:0/1286660021 >> v2:172.21.15.203:3300/0 conn(0x7feac81415d0 0x7feac815eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.893+0000 7feaced8d640 1 --2- 172.21.15.156:0/1286660021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac80b41e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7feac80b4b50 tx=0x7feac002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253ac9a2582cd850 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.893+0000 7feacd58a640 1 -- 172.21.15.156:0/1286660021 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0039070 con 0x7feac80b3dd0 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.893+0000 7feacd58a640 1 -- 172.21.15.156:0/1286660021 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac002cc80 con 0x7feac80b3dd0 2024-03-29T04:10:49.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.893+0000 7feacd58a640 1 -- 172.21.15.156:0/1286660021 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0034660 con 0x7feac80b3dd0 2024-03-29T04:10:49.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 -- 172.21.15.156:0/1286660021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 msgr2=0x7feac80b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 --2- 172.21.15.156:0/1286660021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac80b41e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7feac80b4b50 tx=0x7feac002bed0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 -- 172.21.15.156:0/1286660021 shutdown_connections 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 --2- 172.21.15.156:0/1286660021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac80b41e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 --2- 172.21.15.156:0/1286660021 >> v2:172.21.15.203:3300/0 conn(0x7feac81415d0 0x7feac815eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 -- 172.21.15.156:0/1286660021 >> 172.21.15.156:0/1286660021 conn(0x7feac80b7660 msgr2=0x7feac80b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 -- 172.21.15.156:0/1286660021 shutdown_connections 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.894+0000 7fead07f0640 1 -- 172.21.15.156:0/1286660021 wait complete. 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.895+0000 7fead07f0640 1 Processor -- start 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.895+0000 7fead07f0640 1 -- start start 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac8145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac81415d0 0x7feac80d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7feac8144650 0x7feac80cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feac8144ae0 con 0x7feac80b3dd0 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feac815f6b0 con 0x7feac81415d0 2024-03-29T04:10:49.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7feac815f990 con 0x7feac8144650 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac8145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac8145d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45662/0 (socket says 172.21.15.156:45662) 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 -- 172.21.15.156:0/4189481939 learned_addr learned my addr 172.21.15.156:0/4189481939 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feacdd8b640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac81415d0 0x7feac80d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feaced8d640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7feac8144650 0x7feac80cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 -- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7feac8144650 msgr2=0x7feac80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7feac8144650 0x7feac80cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 -- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac81415d0 msgr2=0x7feac80d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac81415d0 0x7feac80d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 -- 172.21.15.156:0/4189481939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac80d08b0 con 0x7feac80b3dd0 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feace58c640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac8145d30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7feac400c7f0 tx=0x7feac400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4015070 con 0x7feac80b3dd0 2024-03-29T04:10:49.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac4007a30 con 0x7feac80b3dd0 2024-03-29T04:10:49.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4007e00 con 0x7feac80b3dd0 2024-03-29T04:10:49.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac80d0b90 con 0x7feac80b3dd0 2024-03-29T04:10:49.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.896+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac80d1150 con 0x7feac80b3dd0 2024-03-29T04:10:49.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.898+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac804f9d0 con 0x7feac80b3dd0 2024-03-29T04:10:49.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.898+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7feac4018070 con 0x7feac80b3dd0 2024-03-29T04:10:49.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.898+0000 7feabeffd640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7feaac06c530 0x7feaac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:49.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.899+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7feac408b630 con 0x7feac80b3dd0 2024-03-29T04:10:49.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.899+0000 7feacdd8b640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7feaac06c530 0x7feaac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:49.880 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.899+0000 7feacdd8b640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7feaac06c530 0x7feaac06e9f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feac8070590 tx=0x7feab8007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:49.882 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:49.903+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7feac4059600 con 0x7feac80b3dd0 2024-03-29T04:10:50.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.085+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feac80d2f60 con 0x7feac80b3dd0 2024-03-29T04:10:50.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.085+0000 7feabeffd640 1 -- 172.21.15.156:0/4189481939 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7feac405d900 con 0x7feac80b3dd0 2024-03-29T04:10:50.065 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:10:50.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.087+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7feaac06c530 msgr2=0x7feaac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.087+0000 7fead07f0640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7feaac06c530 0x7feaac06e9f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feac8070590 tx=0x7feab8007d50 comp rx=0 tx=0).stop 2024-03-29T04:10:50.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.087+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 msgr2=0x7feac8145d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.087+0000 7fead07f0640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac8145d30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7feac400c7f0 tx=0x7feac400cbf0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.087+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 shutdown_connections 2024-03-29T04:10:50.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7feaac06c530 0x7feaac06e9f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7feac8144650 0x7feac80cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac81415d0 0x7feac80d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 --2- 172.21.15.156:0/4189481939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feac80b3dd0 0x7feac8145d30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 >> 172.21.15.156:0/4189481939 conn(0x7feac80b7660 msgr2=0x7feac80b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:50.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 shutdown_connections 2024-03-29T04:10:50.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.088+0000 7fead07f0640 1 -- 172.21.15.156:0/4189481939 wait complete. 2024-03-29T04:10:50.078 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:10:50.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.220+0000 7f5a9ca4a640 1 Processor -- start 2024-03-29T04:10:50.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.220+0000 7f5a9ca4a640 1 -- start start 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9ca4a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5a980ceb30 0x7f5a980d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9ca4a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5a980cef10 0x7f5a9815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9ca4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9815fc30 con 0x7f5a9809b240 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9ca4a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f5a980b3550 con 0x7f5a980ceb30 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9ca4a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5a980b3830 con 0x7f5a980cef10 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5a980ceb30 0x7f5a980d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5a980ceb30 0x7f5a980d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45664/0 (socket says 172.21.15.156:45664) 2024-03-29T04:10:50.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.221+0000 7f5a9659c640 1 -- 172.21.15.156:0/461408962 learned_addr learned my addr 172.21.15.156:0/461408962 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:50.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.222+0000 7f5a95d9b640 1 --2- 172.21.15.156:0/461408962 >> v2:172.21.15.203:3300/0 conn(0x7f5a980cef10 0x7f5a9815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.222+0000 7f5a9659c640 1 -- 172.21.15.156:0/461408962 >> v2:172.21.15.203:3300/0 conn(0x7f5a980cef10 msgr2=0x7f5a9815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:50.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.222+0000 7f5a9659c640 1 --2- 172.21.15.156:0/461408962 >> v2:172.21.15.203:3300/0 conn(0x7f5a980cef10 0x7f5a9815f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.222+0000 7f5a9659c640 1 -- 172.21.15.156:0/461408962 >> v1:172.21.15.203:6789/0 conn(0x7f5a9809b240 legacy=0x7f5a980b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:50.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.222+0000 7f5a9659c640 1 -- 172.21.15.156:0/461408962 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a980b3b10 con 0x7f5a980ceb30 2024-03-29T04:10:50.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.222+0000 7f5a9659c640 1 --2- 172.21.15.156:0/461408962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a980ceb30 0x7f5a980d1be0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806f7d0 tx=0x7f5a9002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954eabdf6ab371cd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:50.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9559a640 1 -- 172.21.15.156:0/461408962 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9002c9f0 con 0x7f5a980ceb30 2024-03-29T04:10:50.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9559a640 1 -- 172.21.15.156:0/461408962 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9002cb90 con 0x7f5a980ceb30 2024-03-29T04:10:50.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9559a640 1 -- 172.21.15.156:0/461408962 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a900343f0 con 0x7f5a980ceb30 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/461408962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a980ceb30 msgr2=0x7f5a980d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/461408962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a980ceb30 0x7f5a980d1be0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806f7d0 tx=0x7f5a9002be50 comp rx=0 tx=0).stop 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/461408962 shutdown_connections 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/461408962 >> v2:172.21.15.203:3300/0 conn(0x7f5a980cef10 0x7f5a9815f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/461408962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a980ceb30 0x7f5a980d1be0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/461408962 >> 172.21.15.156:0/461408962 conn(0x7f5a980b3df0 msgr2=0x7f5a980b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/461408962 shutdown_connections 2024-03-29T04:10:50.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.223+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/461408962 wait complete. 2024-03-29T04:10:50.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 Processor -- start 2024-03-29T04:10:50.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 -- start start 2024-03-29T04:10:50.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 0x7f5a9815de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5a980ceb30 0x7f5a9814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a980cef10 0x7f5a980b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5a980b3260 con 0x7f5a9809b240 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9809ad20 con 0x7f5a980cef10 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9ca4a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5a980cf3c0 con 0x7f5a980ceb30 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 0x7f5a9815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 0x7f5a9815de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45676/0 (socket says 172.21.15.156:45676) 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.225+0000 7f5a9659c640 1 -- 172.21.15.156:0/1803097337 learned_addr learned my addr 172.21.15.156:0/1803097337 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a96d9d640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a980cef10 0x7f5a980b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a95d9b640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5a980ceb30 0x7f5a9814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9659c640 1 -- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5a980ceb30 msgr2=0x7f5a9814ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9659c640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5a980ceb30 0x7f5a9814ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9659c640 1 -- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a980cef10 msgr2=0x7f5a980b6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9659c640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a980cef10 0x7f5a980b6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9659c640 1 -- 172.21.15.156:0/1803097337 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a980b7260 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9659c640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 0x7f5a9815de10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5a90000f80 tx=0x7f5a90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90043020 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9003e450 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a98158dd0 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90009510 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.226+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a98159390 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.227+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a980a0450 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.228+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5a90046020 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.228+0000 7f5a8effd640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a7806c530 0x7f5a7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.228+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f5a900ade10 con 0x7f5a9809b240 2024-03-29T04:10:50.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.228+0000 7f5a95d9b640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a7806c530 0x7f5a7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.229+0000 7f5a95d9b640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a7806c530 0x7f5a7806e9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a980b44d0 tx=0x7f5a80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:50.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.233+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5a9004d030 con 0x7f5a9809b240 2024-03-29T04:10:50.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.437+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a9804f9d0 con 0x7f5a9809b240 2024-03-29T04:10:50.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.438+0000 7f5a8effd640 1 -- 172.21.15.156:0/1803097337 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7f5a9007c4c0 con 0x7f5a9809b240 2024-03-29T04:10:50.417 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.439+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a7806c530 msgr2=0x7f5a7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.439+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a7806c530 0x7f5a7806e9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a980b44d0 tx=0x7f5a80008040 comp rx=0 tx=0).stop 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.439+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 msgr2=0x7f5a9815de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.439+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 0x7f5a9815de10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5a90000f80 tx=0x7f5a90004a40 comp rx=0 tx=0).stop 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 shutdown_connections 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a7806c530 0x7f5a7806e9f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a980cef10 0x7f5a980b6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5a980ceb30 0x7f5a9814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 --2- 172.21.15.156:0/1803097337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a9809b240 0x7f5a9815de10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 >> 172.21.15.156:0/1803097337 conn(0x7f5a980b3df0 msgr2=0x7f5a9806c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:50.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 shutdown_connections 2024-03-29T04:10:50.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.440+0000 7f5a9ca4a640 1 -- 172.21.15.156:0/1803097337 wait complete. 2024-03-29T04:10:50.432 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-03-29T04:10:50.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:50.432 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-29T04:10:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi156.stderr:ceph-fuse[51774]: fuse finished with error 0 and tester_r 0 2024-03-29T04:10:50.476 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:10:50.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:50.476 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T04:10:50.645 INFO:tasks.cephfs.mount:Terminating background process 2024-03-29T04:10:50.645 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:10:50.645 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:10:50.664 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.809+0000 7fc54153d640 1 Processor -- start 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.809+0000 7fc54153d640 1 -- start start 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc54153d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc53c0b1850 0x7fc53c0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc54153d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 0x7fc53c09aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc54153d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc53c0b1d00 con 0x7fc53c15fba0 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc54153d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc53c0b1fe0 con 0x7fc53c0b1850 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc54153d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc53c0b22c0 con 0x7fc53c0b32e0 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53affd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc53c15fba0 0x7fc53c160a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:38994/0 (socket says 172.21.15.156:38994) 2024-03-29T04:10:50.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53affd640 1 -- 172.21.15.156:0/2101394768 learned_addr learned my addr 172.21.15.156:0/2101394768 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc533fff640 1 --2- 172.21.15.156:0/2101394768 >> v2:172.21.15.203:3300/0 conn(0x7fc53c0b1850 0x7fc53c0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53b7fe640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 0x7fc53c09aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2615439548 0 0) 0x7fc53c0b1d00 con 0x7fc53c15fba0 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc528003520 con 0x7fc53c15fba0 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1894004823 0 0) 0x7fc528003520 con 0x7fc53c15fba0 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 msgr2=0x7fc53c09aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 0x7fc53c09aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 >> v2:172.21.15.203:3300/0 conn(0x7fc53c0b1850 msgr2=0x7fc53c0b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 --2- 172.21.15.156:0/2101394768 >> v2:172.21.15.203:3300/0 conn(0x7fc53c0b1850 0x7fc53c0b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53c0b25a0 con 0x7fc53c15fba0 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fc52c001670 con 0x7fc53c15fba0 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 >> v1:172.21.15.156:6789/0 conn(0x7fc53c15fba0 legacy=0x7fc53c160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 0x7fc5280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc528003f30 0x7fc528004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.810+0000 7fc53a7fc640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc528004d40 0x7fc5280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c0b1fe0 con 0x7fc528004d40 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c0b22c0 con 0x7fc528003f30 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc52800ba10 con 0x7fc53c0b32e0 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53b7fe640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc528004d40 0x7fc5280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53affd640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 0x7fc5280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc533fff640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc528003f30 0x7fc528004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53b7fe640 1 -- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 msgr2=0x7fc5280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:50.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53b7fe640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c0b32e0 0x7fc5280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53b7fe640 1 -- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc528003f30 msgr2=0x7fc528004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:50.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53b7fe640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc528003f30 0x7fc528004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.811+0000 7fc53b7fe640 1 -- 172.21.15.156:0/2101394768 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc528003520 con 0x7fc528004d40 2024-03-29T04:10:50.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc53b7fe640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc528004d40 0x7fc5280051c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc53400e170 tx=0x7fc528005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8ffca5e3eb5e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:50.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc534039070 con 0x7fc528004d40 2024-03-29T04:10:50.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc53402cbf0 con 0x7fc528004d40 2024-03-29T04:10:50.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc53a7fc640 1 -- 172.21.15.156:0/2101394768 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc534009c10 con 0x7fc528004d40 2024-03-29T04:10:50.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc54153d640 1 -- 172.21.15.156:0/2101394768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc528004d40 msgr2=0x7fc5280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc54153d640 1 --2- 172.21.15.156:0/2101394768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc528004d40 0x7fc5280051c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc53400e170 tx=0x7fc528005830 comp rx=0 tx=0).stop 2024-03-29T04:10:50.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc53b7fe640 1 -- 172.21.15.156:0/2101394768 reap_dead start 2024-03-29T04:10:50.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc54153d640 1 -- 172.21.15.156:0/2101394768 shutdown_connections 2024-03-29T04:10:50.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc54153d640 1 -- 172.21.15.156:0/2101394768 >> 172.21.15.156:0/2101394768 conn(0x7fc53c0b3d90 msgr2=0x7fc53c0b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:50.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc54153d640 1 -- 172.21.15.156:0/2101394768 shutdown_connections 2024-03-29T04:10:50.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.812+0000 7fc54153d640 1 -- 172.21.15.156:0/2101394768 wait complete. 2024-03-29T04:10:50.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 Processor -- start 2024-03-29T04:10:50.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 -- start start 2024-03-29T04:10:50.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc53c0b74a0 0x7fc53c1f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 0x7fc53c1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c151e50 0x7fc53c152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c09aa40 con 0x7fc53c0b7880 2024-03-29T04:10:50.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c0b3990 con 0x7fc53c0b74a0 2024-03-29T04:10:50.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc54153d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc53c09abc0 con 0x7fc53c151e50 2024-03-29T04:10:50.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc533fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 0x7fc53c1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc533fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 0x7fc53c1f2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45688/0 (socket says 172.21.15.156:45688) 2024-03-29T04:10:50.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc533fff640 1 -- 172.21.15.156:0/857956797 learned_addr learned my addr 172.21.15.156:0/857956797 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:50.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.814+0000 7fc53affd640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc53c0b74a0 0x7fc53c1f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc53b7fe640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c151e50 0x7fc53c152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.805 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc533fff640 1 -- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c151e50 msgr2=0x7fc53c152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc533fff640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c151e50 0x7fc53c152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc533fff640 1 -- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc53c0b74a0 msgr2=0x7fc53c1f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc533fff640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc53c0b74a0 0x7fc53c1f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc533fff640 1 -- 172.21.15.156:0/857956797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53c1f2a60 con 0x7fc53c0b7880 2024-03-29T04:10:50.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc533fff640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 0x7fc53c1f2210 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc524011600 tx=0x7fc524011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524015870 con 0x7fc53c0b7880 2024-03-29T04:10:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc524015a10 con 0x7fc53c0b7880 2024-03-29T04:10:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53c1f2d40 con 0x7fc53c0b7880 2024-03-29T04:10:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524015dc0 con 0x7fc53c0b7880 2024-03-29T04:10:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.815+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53c1f3600 con 0x7fc53c0b7880 2024-03-29T04:10:50.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.816+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53c1f3420 con 0x7fc53c0b7880 2024-03-29T04:10:50.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.817+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc52400f460 con 0x7fc53c0b7880 2024-03-29T04:10:50.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.818+0000 7fc5337fe640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc51c06c560 0x7fc51c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:50.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.818+0000 7fc53affd640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc51c06c560 0x7fc51c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:50.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.818+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc52408bf40 con 0x7fc53c0b7880 2024-03-29T04:10:50.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.818+0000 7fc53affd640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc51c06c560 0x7fc51c06ea20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc52c004840 tx=0x7fc52c007970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:50.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:50.822+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc53c1f3420 con 0x7fc53c0b7880 2024-03-29T04:10:50.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.004+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc53c068d20 con 0x7fc53c0b7880 2024-03-29T04:10:50.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.005+0000 7fc5337fe640 1 -- 172.21.15.156:0/857956797 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7fc524019030 con 0x7fc53c0b7880 2024-03-29T04:10:50.984 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:50.985 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":34,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:55.015332+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc51c06c560 msgr2=0x7fc51c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc51c06c560 0x7fc51c06ea20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc52c004840 tx=0x7fc52c007970 comp rx=0 tx=0).stop 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 msgr2=0x7fc53c1f2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 0x7fc53c1f2210 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc524011600 tx=0x7fc524011a00 comp rx=0 tx=0).stop 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 shutdown_connections 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc51c06c560 0x7fc51c06ea20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc53c151e50 0x7fc53c152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc53c0b7880 0x7fc53c1f2210 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 --2- 172.21.15.156:0/857956797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc53c0b74a0 0x7fc53c1f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 >> 172.21.15.156:0/857956797 conn(0x7fc53c0b3d90 msgr2=0x7fc53c0b6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 shutdown_connections 2024-03-29T04:10:50.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.007+0000 7fc54153d640 1 -- 172.21.15.156:0/857956797 wait complete. 2024-03-29T04:10:50.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:09:51.822722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:09:52.315661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:10:50.998 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:10:51.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.124+0000 7f9430b38640 1 Processor -- start 2024-03-29T04:10:51.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.124+0000 7f9430b38640 1 -- start start 2024-03-29T04:10:51.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f9430b38640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f9430b38640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f942c0cf340 0x7f942c0d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f9430b38640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f942c15cb10 con 0x7f942c145c70 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f9430b38640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f942c15cdf0 con 0x7f942c0cf340 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f9430b38640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f942c15d0d0 con 0x7f942c0b44e0 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f942ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f942c0cf340 0x7f942c0d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f942ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f942c0cf340 0x7f942c0d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45704/0 (socket says 172.21.15.156:45704) 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.125+0000 7f942ad9d640 1 -- 172.21.15.156:0/3902905673 learned_addr learned my addr 172.21.15.156:0/3902905673 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.126+0000 7f9429d9b640 1 --2- 172.21.15.156:0/3902905673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.126+0000 7f942ad9d640 1 -- 172.21.15.156:0/3902905673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 msgr2=0x7f942c0cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.126+0000 7f942ad9d640 1 --2- 172.21.15.156:0/3902905673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c0cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.126+0000 7f942ad9d640 1 -- 172.21.15.156:0/3902905673 >> v1:172.21.15.203:6789/0 conn(0x7f942c145c70 legacy=0x7f942c0b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.126+0000 7f942ad9d640 1 -- 172.21.15.156:0/3902905673 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c15d3b0 con 0x7f942c0cf340 2024-03-29T04:10:51.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f942ad9d640 1 --2- 172.21.15.156:0/3902905673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c0d0ef0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9424009e50 tx=0x7f942402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c95f57b603064c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f942959a640 1 -- 172.21.15.156:0/3902905673 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942402ca70 con 0x7f942c0cf340 2024-03-29T04:10:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f942959a640 1 -- 172.21.15.156:0/3902905673 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942402cc10 con 0x7f942c0cf340 2024-03-29T04:10:51.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f942959a640 1 -- 172.21.15.156:0/3902905673 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94240343f0 con 0x7f942c0cf340 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 -- 172.21.15.156:0/3902905673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 msgr2=0x7f942c0d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 --2- 172.21.15.156:0/3902905673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c0d0ef0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9424009e50 tx=0x7f942402bed0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 -- 172.21.15.156:0/3902905673 shutdown_connections 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 --2- 172.21.15.156:0/3902905673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c0d0ef0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 --2- 172.21.15.156:0/3902905673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c0cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 -- 172.21.15.156:0/3902905673 >> 172.21.15.156:0/3902905673 conn(0x7f942c0b7660 msgr2=0x7f942c0b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 -- 172.21.15.156:0/3902905673 shutdown_connections 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.127+0000 7f9430b38640 1 -- 172.21.15.156:0/3902905673 wait complete. 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 Processor -- start 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 -- start start 2024-03-29T04:10:51.111 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c06da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c1f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f942c145c70 0x7f942c133a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f942c1461d0 con 0x7f942c0cf340 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f942c146050 con 0x7f942c145c70 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.129+0000 7f9430b38640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f942c0d1430 con 0x7f942c0b44e0 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c1f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c1f7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45712/0 (socket says 172.21.15.156:45712) 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 -- 172.21.15.156:0/2646407358 learned_addr learned my addr 172.21.15.156:0/2646407358 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f942a59c640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c06da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f942ad9d640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f942c145c70 0x7f942c133a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 -- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 msgr2=0x7f942c06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c06da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 -- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f942c145c70 msgr2=0x7f942c133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.112 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f942c145c70 0x7f942c133a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 -- 172.21.15.156:0/2646407358 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c06e280 con 0x7f942c0cf340 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9429d9b640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c1f7290 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f942000e9e0 tx=0x7f942000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420017070 con 0x7f942c0cf340 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942000c9d0 con 0x7f942c0cf340 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c06e560 con 0x7f942c0cf340 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942000cdc0 con 0x7f942c0cf340 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.130+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c0b2a30 con 0x7f942c0cf340 2024-03-29T04:10:51.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.131+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c04f9d0 con 0x7f942c0cf340 2024-03-29T04:10:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.132+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f942001a070 con 0x7f942c0cf340 2024-03-29T04:10:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.133+0000 7f941affd640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f940c06c530 0x7f940c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.133+0000 7f942a59c640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f940c06c530 0x7f940c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.133+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f942008b970 con 0x7f942c0cf340 2024-03-29T04:10:51.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.133+0000 7f942a59c640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f940c06c530 0x7f940c06e9f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f941c0016e0 tx=0x7f941c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.116 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.137+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f94200599d0 con 0x7f942c0cf340 2024-03-29T04:10:51.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.428+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f942c0d2f60 con 0x7f942c0cf340 2024-03-29T04:10:51.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.429+0000 7f941affd640 1 -- 172.21.15.156:0/2646407358 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f942005dcd0 con 0x7f942c0cf340 2024-03-29T04:10:51.408 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:10:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f940c06c530 msgr2=0x7f940c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f940c06c530 0x7f940c06e9f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f941c0016e0 tx=0x7f941c009040 comp rx=0 tx=0).stop 2024-03-29T04:10:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 msgr2=0x7f942c1f7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c1f7290 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f942000e9e0 tx=0x7f942000eea0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 shutdown_connections 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f940c06c530 0x7f940c06e9f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f942c145c70 0x7f942c133a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c0cf340 0x7f942c1f7290 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 --2- 172.21.15.156:0/2646407358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f942c0b44e0 0x7f942c06da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 >> 172.21.15.156:0/2646407358 conn(0x7f942c0b7660 msgr2=0x7f942c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 shutdown_connections 2024-03-29T04:10:51.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.431+0000 7f9430b38640 1 -- 172.21.15.156:0/2646407358 wait complete. 2024-03-29T04:10:51.421 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:10:51.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.547+0000 7f4830d90640 1 Processor -- start 2024-03-29T04:10:51.527 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.547+0000 7f4830d90640 1 -- start start 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f4830d90640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f482c0cfc30 0x7f482c0d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f4830d90640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482c0d2df0 con 0x7f482c0d2120 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f4830d90640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482c0cf190 con 0x7f482c15e920 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f4830d90640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f482c0cf7a0 con 0x7f482c0cfc30 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f482ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f482c0cfc30 0x7f482c0d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f4829d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f482c0d2120 0x7f482c06c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39724/0 (socket says 172.21.15.156:39724) 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f4829d9b640 1 -- 172.21.15.156:0/2749909485 learned_addr learned my addr 172.21.15.156:0/2749909485 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f482ad9d640 1 -- 172.21.15.156:0/2749909485 >> v1:172.21.15.203:6789/0 conn(0x7f482c15e920 legacy=0x7f482c0d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f482ad9d640 1 -- 172.21.15.156:0/2749909485 >> v1:172.21.15.156:6789/0 conn(0x7f482c0d2120 legacy=0x7f482c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f482ad9d640 1 -- 172.21.15.156:0/2749909485 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c0d3080 con 0x7f482c0cfc30 2024-03-29T04:10:51.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.548+0000 7f482959a640 1 -- 172.21.15.156:0/2749909485 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2213686756 0 0) 0x7f482c0d2df0 con 0x7f482c0d2120 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.549+0000 7f482ad9d640 1 --2- 172.21.15.156:0/2749909485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0cfc30 0x7f482c0d00c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f481c0097a0 tx=0x7f481c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6371f0688cf6a5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.549+0000 7f482959a640 1 -- 172.21.15.156:0/2749909485 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c034070 con 0x7f482c0cfc30 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.549+0000 7f482959a640 1 -- 172.21.15.156:0/2749909485 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c030610 con 0x7f482c0cfc30 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.549+0000 7f482959a640 1 -- 172.21.15.156:0/2749909485 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c0309e0 con 0x7f482c0cfc30 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 -- 172.21.15.156:0/2749909485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0cfc30 msgr2=0x7f482c0d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 --2- 172.21.15.156:0/2749909485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0cfc30 0x7f482c0d00c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f481c0097a0 tx=0x7f481c02b960 comp rx=0 tx=0).stop 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 -- 172.21.15.156:0/2749909485 shutdown_connections 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 --2- 172.21.15.156:0/2749909485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0cfc30 0x7f482c0d00c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 -- 172.21.15.156:0/2749909485 >> 172.21.15.156:0/2749909485 conn(0x7f482c09d7a0 msgr2=0x7f482c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 -- 172.21.15.156:0/2749909485 shutdown_connections 2024-03-29T04:10:51.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.550+0000 7f4830d90640 1 -- 172.21.15.156:0/2749909485 wait complete. 2024-03-29T04:10:51.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.552+0000 7f4830d90640 1 Processor -- start 2024-03-29T04:10:51.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.552+0000 7f4830d90640 1 -- start start 2024-03-29T04:10:51.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.553+0000 7f4830d90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f482c0cfc30 0x7f482c0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.553+0000 7f4830d90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 0x7f482c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.553+0000 7f4830d90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f482c15e920 0x7f482c0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.553+0000 7f4830d90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f482c06cce0 con 0x7f482c0d2120 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.553+0000 7f4830d90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f482c0d3360 con 0x7f482c0cfc30 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4830d90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f482c0cdae0 con 0x7f482c15e920 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 0x7f482c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 0x7f482c0b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38050/0 (socket says 172.21.15.156:38050) 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 -- 172.21.15.156:0/1843978252 learned_addr learned my addr 172.21.15.156:0/1843978252 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f482a59c640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f482c0cfc30 0x7f482c0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f482ad9d640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f482c15e920 0x7f482c0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 -- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f482c15e920 msgr2=0x7f482c0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f482c15e920 0x7f482c0b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 -- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f482c0cfc30 msgr2=0x7f482c0cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f482c0cfc30 0x7f482c0cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 -- 172.21.15.156:0/1843978252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c0c8280 con 0x7f482c0d2120 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f482ad9d640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f482c15e920 0x7f482c0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4829d9b640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 0x7f482c0b2670 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f482000c780 tx=0x7f482000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820018070 con 0x7f482c0d2120 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482000a640 con 0x7f482c0d2120 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482c0c8560 con 0x7f482c0d2120 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482000aab0 con 0x7f482c0d2120 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.554+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f482c0c8b20 con 0x7f482c0d2120 2024-03-29T04:10:51.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.555+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f482c0a2480 con 0x7f482c0d2120 2024-03-29T04:10:51.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.556+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f482001b070 con 0x7f482c0d2120 2024-03-29T04:10:51.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.556+0000 7f481affd640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f480c06c530 0x7f480c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.557+0000 7f482a59c640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f480c06c530 0x7f480c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.557+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f482008acd0 con 0x7f482c0d2120 2024-03-29T04:10:51.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.557+0000 7f482a59c640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f480c06c530 0x7f480c06e9f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4824000f60 tx=0x7f4824009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.540 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.561+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f48200592f0 con 0x7f482c0d2120 2024-03-29T04:10:51.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.742+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f482c04f9d0 con 0x7f482c0d2120 2024-03-29T04:10:51.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.742+0000 7f481affd640 1 -- 172.21.15.156:0/1843978252 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f4820058bf0 con 0x7f482c0d2120 2024-03-29T04:10:51.721 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f480c06c530 msgr2=0x7f480c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f480c06c530 0x7f480c06e9f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4824000f60 tx=0x7f4824009040 comp rx=0 tx=0).stop 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 msgr2=0x7f482c0b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 0x7f482c0b2670 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f482000c780 tx=0x7f482000cc40 comp rx=0 tx=0).stop 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 shutdown_connections 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f480c06c530 0x7f480c06e9f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f482c15e920 0x7f482c0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f482c0d2120 0x7f482c0b2670 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 --2- 172.21.15.156:0/1843978252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f482c0cfc30 0x7f482c0cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 >> 172.21.15.156:0/1843978252 conn(0x7f482c09d7a0 msgr2=0x7f482c0b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 shutdown_connections 2024-03-29T04:10:51.724 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.744+0000 7f4830d90640 1 -- 172.21.15.156:0/1843978252 wait complete. 2024-03-29T04:10:51.734 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:10:51.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.860+0000 7f0137ed3640 1 Processor -- start 2024-03-29T04:10:51.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.861+0000 7f0137ed3640 1 -- start start 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0137ed3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0137ed3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f01300ae150 0x7f0130166730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0137ed3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013015db80 con 0x7f013006b5b0 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0137ed3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f013015d650 con 0x7f01300ae150 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0137ed3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f01300b3960 con 0x7f01300b6480 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0136470640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f01300ae150 0x7f0130166730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0136470640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f01300ae150 0x7f0130166730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38052/0 (socket says 172.21.15.156:38052) 2024-03-29T04:10:51.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f0136470640 1 -- 172.21.15.156:0/1484469157 learned_addr learned my addr 172.21.15.156:0/1484469157 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:51.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.862+0000 7f013546e640 1 --2- 172.21.15.156:0/1484469157 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.863+0000 7f0136470640 1 -- 172.21.15.156:0/1484469157 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 msgr2=0x7f01301641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:51.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.863+0000 7f0136470640 1 --2- 172.21.15.156:0/1484469157 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301641e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.863+0000 7f0136470640 1 -- 172.21.15.156:0/1484469157 >> v1:172.21.15.203:6789/0 conn(0x7f013006b5b0 legacy=0x7f013006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.863+0000 7f0136470640 1 -- 172.21.15.156:0/1484469157 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013009abe0 con 0x7f01300ae150 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.863+0000 7f0136470640 1 --2- 172.21.15.156:0/1484469157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01300ae150 0x7f0130166730 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0128009e50 tx=0x7f012802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4b115af797b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.863+0000 7f0134c6d640 1 -- 172.21.15.156:0/1484469157 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012802c9e0 con 0x7f01300ae150 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0134c6d640 1 -- 172.21.15.156:0/1484469157 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f012802cb80 con 0x7f01300ae150 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0134c6d640 1 -- 172.21.15.156:0/1484469157 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01280343f0 con 0x7f01300ae150 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 -- 172.21.15.156:0/1484469157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01300ae150 msgr2=0x7f0130166730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 --2- 172.21.15.156:0/1484469157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01300ae150 0x7f0130166730 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0128009e50 tx=0x7f012802be40 comp rx=0 tx=0).stop 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 -- 172.21.15.156:0/1484469157 shutdown_connections 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 --2- 172.21.15.156:0/1484469157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01300ae150 0x7f0130166730 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 --2- 172.21.15.156:0/1484469157 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301641e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 -- 172.21.15.156:0/1484469157 >> 172.21.15.156:0/1484469157 conn(0x7f0130072d10 msgr2=0x7f01300b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:51.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 -- 172.21.15.156:0/1484469157 shutdown_connections 2024-03-29T04:10:51.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.864+0000 7f0137ed3640 1 -- 172.21.15.156:0/1484469157 wait complete. 2024-03-29T04:10:51.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 Processor -- start 2024-03-29T04:10:51.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 -- start start 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 0x7f0130143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01300ae150 0x7f013013f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0130146110 con 0x7f013006b5b0 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f013006c3d0 con 0x7f01300ae150 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0137ed3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f0130146290 con 0x7f01300b6480 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0135c6f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 0x7f0130143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0135c6f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 0x7f0130143850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38056/0 (socket says 172.21.15.156:38056) 2024-03-29T04:10:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.866+0000 7f0135c6f640 1 -- 172.21.15.156:0/2494789182 learned_addr learned my addr 172.21.15.156:0/2494789182 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f013546e640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01300ae150 0x7f013013f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0136470640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0135c6f640 1 -- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 msgr2=0x7f01301f5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0135c6f640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301f5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0135c6f640 1 -- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01300ae150 msgr2=0x7f013013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0135c6f640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01300ae150 0x7f013013f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0135c6f640 1 -- 172.21.15.156:0/2494789182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01301f6860 con 0x7f013006b5b0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0136470640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0135c6f640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 0x7f0130143850 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c400 tx=0x7f012c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c017070 con 0x7f013006b5b0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f012c0052d0 con 0x7f013006b5b0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01301f6b40 con 0x7f013006b5b0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c003830 con 0x7f013006b5b0 2024-03-29T04:10:51.850 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.867+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0130143050 con 0x7f013006b5b0 2024-03-29T04:10:51.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.868+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013009f4d0 con 0x7f013006b5b0 2024-03-29T04:10:51.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.869+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f012c01a070 con 0x7f013006b5b0 2024-03-29T04:10:51.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.869+0000 7f01267fc640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f011406c530 0x7f011406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:51.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.869+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f012c08a990 con 0x7f013006b5b0 2024-03-29T04:10:51.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.869+0000 7f013546e640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f011406c530 0x7f011406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:51.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.870+0000 7f013546e640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f011406c530 0x7f011406e9f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f01301424e0 tx=0x7f0120009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:51.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:51.873+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f012c058960 con 0x7f013006b5b0 2024-03-29T04:10:52.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.053+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f01300cf390 con 0x7f013006b5b0 2024-03-29T04:10:52.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.054+0000 7f01267fc640 1 -- 172.21.15.156:0/2494789182 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7f012c05cc60 con 0x7f013006b5b0 2024-03-29T04:10:52.033 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f011406c530 msgr2=0x7f011406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f011406c530 0x7f011406e9f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f01301424e0 tx=0x7f0120009190 comp rx=0 tx=0).stop 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 msgr2=0x7f0130143850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 0x7f0130143850 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c400 tx=0x7f012c00c8c0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 shutdown_connections 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f011406c530 0x7f011406e9f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01300b6480 0x7f01301f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01300ae150 0x7f013013f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 --2- 172.21.15.156:0/2494789182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f013006b5b0 0x7f0130143850 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 >> 172.21.15.156:0/2494789182 conn(0x7f0130072d10 msgr2=0x7f013006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:52.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 shutdown_connections 2024-03-29T04:10:52.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.056+0000 7f0137ed3640 1 -- 172.21.15.156:0/2494789182 wait complete. 2024-03-29T04:10:52.046 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T04:10:52.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.170+0000 7f79830f2640 1 Processor -- start 2024-03-29T04:10:52.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.170+0000 7f79830f2640 1 -- start start 2024-03-29T04:10:52.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f79830f2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f797c15e920 0x7f797c0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f79830f2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c0d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f79830f2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c0cf190 con 0x7f797c0d2120 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f79830f2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f797c0cf7a0 con 0x7f797c15e920 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f79830f2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f797c0d3080 con 0x7f797c0cfc30 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f797c15e920 0x7f797c0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f797c15e920 0x7f797c0d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38072/0 (socket says 172.21.15.156:38072) 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 -- 172.21.15.156:0/2917800396 learned_addr learned my addr 172.21.15.156:0/2917800396 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f798168f640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c0d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 -- 172.21.15.156:0/2917800396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 msgr2=0x7f797c0d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c0d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.152 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 -- 172.21.15.156:0/2917800396 >> v1:172.21.15.203:6789/0 conn(0x7f797c0d2120 legacy=0x7f797c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f7980e8e640 1 -- 172.21.15.156:0/2917800396 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c0cf430 con 0x7f797c15e920 2024-03-29T04:10:52.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.171+0000 7f798168f640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c0d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:52.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.172+0000 7f7980e8e640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c15e920 0x7f797c0d26e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7970009870 tx=0x7f797002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a27a36e7848f63 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:52.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.172+0000 7f797b7fe640 1 -- 172.21.15.156:0/2917800396 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7970034070 con 0x7f797c15e920 2024-03-29T04:10:52.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.172+0000 7f797b7fe640 1 -- 172.21.15.156:0/2917800396 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7970030840 con 0x7f797c15e920 2024-03-29T04:10:52.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.172+0000 7f797b7fe640 1 -- 172.21.15.156:0/2917800396 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7970030c10 con 0x7f797c15e920 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 -- 172.21.15.156:0/2917800396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c15e920 msgr2=0x7f797c0d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c15e920 0x7f797c0d26e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7970009870 tx=0x7f797002b980 comp rx=0 tx=0).stop 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 -- 172.21.15.156:0/2917800396 shutdown_connections 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c0d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 --2- 172.21.15.156:0/2917800396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c15e920 0x7f797c0d26e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 -- 172.21.15.156:0/2917800396 >> 172.21.15.156:0/2917800396 conn(0x7f797c09d7a0 msgr2=0x7f797c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 -- 172.21.15.156:0/2917800396 shutdown_connections 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.173+0000 7f79830f2640 1 -- 172.21.15.156:0/2917800396 wait complete. 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.174+0000 7f79830f2640 1 Processor -- start 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.174+0000 7f79830f2640 1 -- start start 2024-03-29T04:10:52.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f79830f2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f79830f2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 0x7f797c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f79830f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f797c15e920 0x7f797c0cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f79830f2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f797c0d34e0 con 0x7f797c0d2120 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f79830f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f797c0d3360 con 0x7f797c15e920 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f79830f2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f797c0d2c20 con 0x7f797c0cfc30 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 0x7f797c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 0x7f797c0cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38088/0 (socket says 172.21.15.156:38088) 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 -- 172.21.15.156:0/2166203258 learned_addr learned my addr 172.21.15.156:0/2166203258 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 -- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 msgr2=0x7f797c15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 -- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f797c15e920 msgr2=0x7f797c0cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f798168f640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f797c15e920 0x7f797c0cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f797c15e920 0x7f797c0cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 -- 172.21.15.156:0/2166203258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c1e9920 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.175+0000 7f797bfff640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 0x7f797c0cb5e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f796800c400 tx=0x7f796800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.176+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79680052d0 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.176+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7968005490 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.176+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797c15d240 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.176+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968007260 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.176+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f797c09f720 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.177+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f797c0a2480 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.177+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7968007400 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.178+0000 7f79797fa640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f796006c560 0x7f796006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.178+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f796808aca0 con 0x7f797c0d2120 2024-03-29T04:10:52.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.178+0000 7f7980e8e640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f796006c560 0x7f796006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.178+0000 7f7980e8e640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f796006c560 0x7f796006ea20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7970036000 tx=0x7f7970030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:52.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.182+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7968059230 con 0x7f797c0d2120 2024-03-29T04:10:52.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.368+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f797c04f9d0 con 0x7f797c0d2120 2024-03-29T04:10:52.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.368+0000 7f79797fa640 1 -- 172.21.15.156:0/2166203258 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7f7968058b30 con 0x7f797c0d2120 2024-03-29T04:10:52.347 INFO:teuthology.orchestra.run.smithi156.stderr:listed 0 entries 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f796006c560 msgr2=0x7f796006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f796006c560 0x7f796006ea20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7970036000 tx=0x7f7970030000 comp rx=0 tx=0).stop 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 msgr2=0x7f797c0cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 0x7f797c0cb5e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f796800c400 tx=0x7f796800c8c0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 shutdown_connections 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f796006c560 0x7f796006ea20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f797c15e920 0x7f797c0cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f797c0d2120 0x7f797c0cb5e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 --2- 172.21.15.156:0/2166203258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f797c0cfc30 0x7f797c15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 >> 172.21.15.156:0/2166203258 conn(0x7f797c09d7a0 msgr2=0x7f797c0d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:52.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 shutdown_connections 2024-03-29T04:10:52.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.370+0000 7f79830f2640 1 -- 172.21.15.156:0/2166203258 wait complete. 2024-03-29T04:10:52.361 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi203.front.sepia.ceph.com... 2024-03-29T04:10:52.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:52.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-29T04:10:52.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[44721]: fuse finished with error 0 and tester_r 0 2024-03-29T04:10:52.394 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:10:52.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:52.394 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-29T04:10:52.488 INFO:tasks.cephfs.mount:Terminating background process 2024-03-29T04:10:52.488 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:10:52.488 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:10:52.503 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17587 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-29T04:10:52.504 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:10:52.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:52.504 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-29T04:10:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:10:52.559 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-29T04:10:52.559 INFO:tasks.cephfs.mount:Terminating background process 2024-03-29T04:10:52.559 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:10:52.559 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:10:52.577 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056352,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=427 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-29T04:10:52.577 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:10:52.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:10:52.577 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-29T04:10:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:10:52.635 INFO:teuthology.orchestra.run.smithi203.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-29T04:10:52.636 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:10:52.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.757+0000 7f5494f7c640 1 Processor -- start 2024-03-29T04:10:52.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5494f7c640 1 -- start start 2024-03-29T04:10:52.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5494f7c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f549015e730 0x7f54900ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5494f7c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f54900cf3e0 0x7f54900d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5494f7c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54900cfa20 con 0x7f54900d4800 2024-03-29T04:10:52.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5494f7c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f549006c720 con 0x7f54900cf3e0 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5494f7c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f54900cfd00 con 0x7f549015e730 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f548ed9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f54900d4800 0x7f54900d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39734/0 (socket says 172.21.15.156:39734) 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f5487fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f54900cf3e0 0x7f54900d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.758+0000 7f548ed9d640 1 -- 172.21.15.156:0/197158656 learned_addr learned my addr 172.21.15.156:0/197158656 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f548e59c640 1 --2- 172.21.15.156:0/197158656 >> v2:172.21.15.203:3300/0 conn(0x7f549015e730 0x7f54900ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f5487fff640 1 -- 172.21.15.156:0/197158656 >> v2:172.21.15.203:3300/0 conn(0x7f549015e730 msgr2=0x7f54900ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f5487fff640 1 --2- 172.21.15.156:0/197158656 >> v2:172.21.15.203:3300/0 conn(0x7f549015e730 0x7f54900ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f5487fff640 1 -- 172.21.15.156:0/197158656 >> v1:172.21.15.156:6789/0 conn(0x7f54900d4800 legacy=0x7f54900d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f5487fff640 1 -- 172.21.15.156:0/197158656 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54900cffe0 con 0x7f54900cf3e0 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f548e59c640 1 --2- 172.21.15.156:0/197158656 >> v2:172.21.15.203:3300/0 conn(0x7f549015e730 0x7f54900ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:52.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.759+0000 7f5487fff640 1 --2- 172.21.15.156:0/197158656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54900cf3e0 0x7f54900d42c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f54900d1f20 tx=0x7f547802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2cb635b22c132d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:52.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f548dd9b640 1 -- 172.21.15.156:0/197158656 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478035020 con 0x7f54900cf3e0 2024-03-29T04:10:52.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f548dd9b640 1 -- 172.21.15.156:0/197158656 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5478031830 con 0x7f54900cf3e0 2024-03-29T04:10:52.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f548dd9b640 1 -- 172.21.15.156:0/197158656 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478031c00 con 0x7f54900cf3e0 2024-03-29T04:10:52.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 -- 172.21.15.156:0/197158656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54900cf3e0 msgr2=0x7f54900d42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 --2- 172.21.15.156:0/197158656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54900cf3e0 0x7f54900d42c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f54900d1f20 tx=0x7f547802b8d0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 -- 172.21.15.156:0/197158656 shutdown_connections 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 --2- 172.21.15.156:0/197158656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54900cf3e0 0x7f54900d42c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 --2- 172.21.15.156:0/197158656 >> v2:172.21.15.203:3300/0 conn(0x7f549015e730 0x7f54900ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 -- 172.21.15.156:0/197158656 >> 172.21.15.156:0/197158656 conn(0x7f549008bfd0 msgr2=0x7f549009d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 -- 172.21.15.156:0/197158656 shutdown_connections 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.760+0000 7f5494f7c640 1 -- 172.21.15.156:0/197158656 wait complete. 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.762+0000 7f5494f7c640 1 Processor -- start 2024-03-29T04:10:52.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.762+0000 7f5494f7c640 1 -- start start 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.763+0000 7f5494f7c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54900cf3e0 0x7f5490160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.763+0000 7f5494f7c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54900d4800 0x7f549014a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.763+0000 7f5494f7c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 0x7f54900cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.763+0000 7f5494f7c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54900d2220 con 0x7f549015e730 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.763+0000 7f5494f7c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54900d4c30 con 0x7f54900d4800 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.763+0000 7f5494f7c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f549015eb10 con 0x7f54900cf3e0 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 0x7f54900cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 0x7f54900cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38114/0 (socket says 172.21.15.156:38114) 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 -- 172.21.15.156:0/2125948304 learned_addr learned my addr 172.21.15.156:0/2125948304 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f5487fff640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54900d4800 0x7f549014a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548e59c640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54900cf3e0 0x7f5490160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 -- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54900cf3e0 msgr2=0x7f5490160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54900cf3e0 0x7f5490160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 -- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54900d4800 msgr2=0x7f549014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54900d4800 0x7f549014a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 -- 172.21.15.156:0/2125948304 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5490160d50 con 0x7f549015e730 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f548ed9d640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 0x7f54900cb500 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f548800cbc0 tx=0x7f548800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5488017070 con 0x7f549015e730 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f548800ad40 con 0x7f549015e730 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5488012600 con 0x7f549015e730 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f5494f7c640 1 -- 172.21.15.156:0/2125948304 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549014ae50 con 0x7f549015e730 2024-03-29T04:10:52.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.764+0000 7f5494f7c640 1 -- 172.21.15.156:0/2125948304 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54901612d0 con 0x7f549015e730 2024-03-29T04:10:52.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.766+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5490068610 con 0x7f549015e730 2024-03-29T04:10:52.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.766+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f548801a070 con 0x7f549015e730 2024-03-29T04:10:52.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.766+0000 7f5486ffd640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f547006c530 0x7f547006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:52.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.766+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f548808b370 con 0x7f549015e730 2024-03-29T04:10:52.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.766+0000 7f548e59c640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f547006c530 0x7f547006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:52.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.767+0000 7f548e59c640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f547006c530 0x7f547006e9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f548000a950 tx=0x7f5480008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:52.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:52.775+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5488059340 con 0x7f549015e730 2024-03-29T04:10:53.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.100+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54900b31d0 con 0x7f549015e730 2024-03-29T04:10:53.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.100+0000 7f5486ffd640 1 -- 172.21.15.156:0/2125948304 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5190 (secure 0 0 0) 0x7f548805d640 con 0x7f549015e730 2024-03-29T04:10:53.080 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:53.080 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:09:53.360769+0000","modified":"2024-03-29T04:10:10.696817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5310":{"gid":5310,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/788767406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":788767406},{"type":"v1","addr":"172.21.15.156:6837","nonce":788767406}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5310,"qdb_cluster":[5310]},"id":2}]} 2024-03-29T04:10:53.080 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 17 2024-03-29T04:10:53.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.102+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f547006c530 msgr2=0x7f547006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.102+0000 7f5484ff9640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f547006c530 0x7f547006e9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f548000a950 tx=0x7f5480008040 comp rx=0 tx=0).stop 2024-03-29T04:10:53.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.102+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 msgr2=0x7f54900cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.102+0000 7f5484ff9640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 0x7f54900cb500 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f548800cbc0 tx=0x7f548800a430 comp rx=0 tx=0).stop 2024-03-29T04:10:53.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 shutdown_connections 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f547006c530 0x7f547006e9f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549015e730 0x7f54900cb500 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54900d4800 0x7f549014a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 --2- 172.21.15.156:0/2125948304 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54900cf3e0 0x7f5490160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 >> 172.21.15.156:0/2125948304 conn(0x7f549008bfd0 msgr2=0x7f54900b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 shutdown_connections 2024-03-29T04:10:53.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.103+0000 7f5484ff9640 1 -- 172.21.15.156:0/2125948304 wait complete. 2024-03-29T04:10:53.093 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:10:53.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.216+0000 7f5dcd295640 1 Processor -- start 2024-03-29T04:10:53.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.216+0000 7f5dcd295640 1 -- start start 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dcd295640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5dc815e730 0x7f5dc80ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dcd295640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dc80cfa20 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dcd295640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dc80cfd00 con 0x7f5dc80d49d0 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dcd295640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f5dc806c720 con 0x7f5dc815e730 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dc6ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5dc815e730 0x7f5dc80ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dc67fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5dc80cf3e0 0x7f5dc80d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39736/0 (socket says 172.21.15.156:39736) 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dc6ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5dc815e730 0x7f5dc80ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38130/0 (socket says 172.21.15.156:38130) 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dc67fc640 1 -- 172.21.15.156:0/3060170816 learned_addr learned my addr 172.21.15.156:0/3060170816 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.217+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3060170816 >> v1:172.21.15.203:6789/0 conn(0x7f5dc80d49d0 legacy=0x7f5dc806cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3060170816 >> v1:172.21.15.156:6789/0 conn(0x7f5dc80cf3e0 legacy=0x7f5dc80d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3060170816 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc80cffe0 con 0x7f5dc815e730 2024-03-29T04:10:53.197 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc5ffb640 1 -- 172.21.15.156:0/3060170816 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107804855 0 0) 0x7f5dc80cfa20 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc6ffd640 1 --2- 172.21.15.156:0/3060170816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc815e730 0x7f5dc80ceea0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097a0 tx=0x7f5dbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5f21e52dda54ac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc5ffb640 1 -- 172.21.15.156:0/3060170816 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc035020 con 0x7f5dc815e730 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc5ffb640 1 -- 172.21.15.156:0/3060170816 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc031830 con 0x7f5dc815e730 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.218+0000 7f5dc5ffb640 1 -- 172.21.15.156:0/3060170816 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc031c00 con 0x7f5dc815e730 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 -- 172.21.15.156:0/3060170816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc815e730 msgr2=0x7f5dc80ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3060170816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc815e730 0x7f5dc80ceea0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097a0 tx=0x7f5dbc02b8d0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 -- 172.21.15.156:0/3060170816 shutdown_connections 2024-03-29T04:10:53.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3060170816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc815e730 0x7f5dc80ceea0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 -- 172.21.15.156:0/3060170816 >> 172.21.15.156:0/3060170816 conn(0x7f5dc808bfd0 msgr2=0x7f5dc809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:53.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 -- 172.21.15.156:0/3060170816 shutdown_connections 2024-03-29T04:10:53.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.219+0000 7f5dcd295640 1 -- 172.21.15.156:0/3060170816 wait complete. 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.220+0000 7f5dcd295640 1 Processor -- start 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.220+0000 7f5dcd295640 1 -- start start 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dcd295640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 0x7f5dc80cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dcd295640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dc80d49d0 0x7f5dc80cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dcd295640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc815e730 0x7f5dc8153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dcd295640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc815eb10 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dcd295640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc80d2220 con 0x7f5dc80d49d0 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dcd295640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc815ec90 con 0x7f5dc815e730 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dc6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 0x7f5dc80cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dc6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 0x7f5dc80cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38144/0 (socket says 172.21.15.156:38144) 2024-03-29T04:10:53.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3902070636 learned_addr learned my addr 172.21.15.156:0/3902070636 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dc67fc640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dc80d49d0 0x7f5dc80cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.221+0000 7f5dc77fe640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc815e730 0x7f5dc8153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc815e730 msgr2=0x7f5dc8153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dc6ffd640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc815e730 0x7f5dc8153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dc80d49d0 msgr2=0x7f5dc80cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dc6ffd640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dc80d49d0 0x7f5dc80cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dc6ffd640 1 -- 172.21.15.156:0/3902070636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc80c7dc0 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dc6ffd640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 0x7f5dc80cabe0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc000c00 tx=0x7f5dbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc040070 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc039a00 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc035040 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc80c80a0 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.222+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc80c8660 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.224+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5dbc04b060 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.224+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc8068610 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.224+0000 7f5daf7fe640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5db406c530 0x7f5db406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.224+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f5dbc0acf10 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.204 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.224+0000 7f5dc67fc640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5db406c530 0x7f5db406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.224+0000 7f5dc67fc640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5db406c530 0x7f5db406e9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5db8007fd0 tx=0x7f5db8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.228+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5dbc07af70 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.423 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.444+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5dc80cee60 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.445+0000 7f5daf7fe640 1 -- 172.21.15.156:0/3902070636 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7f5dbc07f270 con 0x7f5dc80cf3e0 2024-03-29T04:10:53.425 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:53.425 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":34,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:55.015332+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:10:53.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5db406c530 msgr2=0x7f5db406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5db406c530 0x7f5db406e9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5db8007fd0 tx=0x7f5db8009040 comp rx=0 tx=0).stop 2024-03-29T04:10:53.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 msgr2=0x7f5dc80cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 0x7f5dc80cabe0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc000c00 tx=0x7f5dbc0046c0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 shutdown_connections 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5db406c530 0x7f5db406e9f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc815e730 0x7f5dc8153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dc80d49d0 0x7f5dc80cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 --2- 172.21.15.156:0/3902070636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc80cf3e0 0x7f5dc80cabe0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 >> 172.21.15.156:0/3902070636 conn(0x7f5dc808bfd0 msgr2=0x7f5dc815dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 shutdown_connections 2024-03-29T04:10:53.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.447+0000 7f5dcd295640 1 -- 172.21.15.156:0/3902070636 wait complete. 2024-03-29T04:10:53.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T04:08:41.651889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T04:09:51.822722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T04:09:52.315661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T04:10:53.439 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T04:10:53.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.564+0000 7f32b116d640 1 Processor -- start 2024-03-29T04:10:53.543 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.564+0000 7f32b116d640 1 -- start start 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32b116d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f32ac070480 0x7f32ac06f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32b116d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac06b5b0 0x7f32ac06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32b116d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ac0b5ed0 con 0x7f32ac15e960 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32b116d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f32ac06bf60 con 0x7f32ac070480 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32b116d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac09aa40 con 0x7f32ac06b5b0 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f32ac070480 0x7f32ac06f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f32ac070480 0x7f32ac06f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38148/0 (socket says 172.21.15.156:38148) 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 -- 172.21.15.156:0/3297131358 learned_addr learned my addr 172.21.15.156:0/3297131358 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32ab59e640 1 --2- 172.21.15.156:0/3297131358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac06b5b0 0x7f32ac06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 -- 172.21.15.156:0/3297131358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac06b5b0 msgr2=0x7f32ac06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 --2- 172.21.15.156:0/3297131358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac06b5b0 0x7f32ac06ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 -- 172.21.15.156:0/3297131358 >> v1:172.21.15.203:6789/0 conn(0x7f32ac15e960 legacy=0x7f32ac0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:53.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.565+0000 7f32aa59c640 1 -- 172.21.15.156:0/3297131358 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac09ac90 con 0x7f32ac070480 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.566+0000 7f32aa59c640 1 --2- 172.21.15.156:0/3297131358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac070480 0x7f32ac06f900 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f329c009870 tx=0x7f329c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da40757812c1cd0c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.566+0000 7f32a9d9b640 1 -- 172.21.15.156:0/3297131358 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329c034070 con 0x7f32ac070480 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.566+0000 7f32a9d9b640 1 -- 172.21.15.156:0/3297131358 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329c030840 con 0x7f32ac070480 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.566+0000 7f32a9d9b640 1 -- 172.21.15.156:0/3297131358 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329c030c10 con 0x7f32ac070480 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 -- 172.21.15.156:0/3297131358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac070480 msgr2=0x7f32ac06f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 --2- 172.21.15.156:0/3297131358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac070480 0x7f32ac06f900 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f329c009870 tx=0x7f329c02b980 comp rx=0 tx=0).stop 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 -- 172.21.15.156:0/3297131358 shutdown_connections 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 --2- 172.21.15.156:0/3297131358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac06b5b0 0x7f32ac06ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 --2- 172.21.15.156:0/3297131358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac070480 0x7f32ac06f900 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 -- 172.21.15.156:0/3297131358 >> 172.21.15.156:0/3297131358 conn(0x7f32ac072ce0 msgr2=0x7f32ac0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 -- 172.21.15.156:0/3297131358 shutdown_connections 2024-03-29T04:10:53.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.567+0000 7f32b116d640 1 -- 172.21.15.156:0/3297131358 wait complete. 2024-03-29T04:10:53.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.568+0000 7f32b116d640 1 Processor -- start 2024-03-29T04:10:53.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.568+0000 7f32b116d640 1 -- start start 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32b116d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32ac06b5b0 0x7f32ac14bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32b116d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac070480 0x7f32ac15f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32b116d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 0x7f32ac161800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32b116d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac0701f0 con 0x7f32ac15e960 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32b116d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac0b6450 con 0x7f32ac06b5b0 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32b116d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac0b49e0 con 0x7f32ac070480 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 0x7f32ac161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 0x7f32ac161800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38152/0 (socket says 172.21.15.156:38152) 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 -- 172.21.15.156:0/2273146146 learned_addr learned my addr 172.21.15.156:0/2273146146 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32aa59c640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac070480 0x7f32ac15f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32aad9d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32ac06b5b0 0x7f32ac14bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 -- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac070480 msgr2=0x7f32ac15f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac070480 0x7f32ac15f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 -- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32ac06b5b0 msgr2=0x7f32ac14bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32ac06b5b0 0x7f32ac14bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 -- 172.21.15.156:0/2273146146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac160500 con 0x7f32ac15e960 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32aad9d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32ac06b5b0 0x7f32ac14bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.569+0000 7f32ab59e640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 0x7f32ac161800 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f32a4007e70 tx=0x7f32a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.570+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4014620 con 0x7f32ac15e960 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.570+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a4019070 con 0x7f32ac15e960 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.570+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ac1607a0 con 0x7f32ac15e960 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.570+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4014a20 con 0x7f32ac15e960 2024-03-29T04:10:53.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.570+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ac160d60 con 0x7f32ac15e960 2024-03-29T04:10:53.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.571+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ac09f3c0 con 0x7f32ac15e960 2024-03-29T04:10:53.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.571+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f32a4014bc0 con 0x7f32ac15e960 2024-03-29T04:10:53.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.571+0000 7f32937fe640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f329806c560 0x7f329806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.572+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f32a408bf50 con 0x7f32ac15e960 2024-03-29T04:10:53.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.572+0000 7f32aad9d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f329806c560 0x7f329806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.572+0000 7f32aad9d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f329806c560 0x7f329806ea20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f32a00015e0 tx=0x7f32a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.576+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f32a405a600 con 0x7f32ac15e960 2024-03-29T04:10:53.730 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.751+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f32ac0ae830 con 0x7f32ac15e960 2024-03-29T04:10:53.730 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.751+0000 7f32937fe640 1 -- 172.21.15.156:0/2273146146 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f32a4059f00 con 0x7f32ac15e960 2024-03-29T04:10:53.731 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-03-29T04:10:53.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f329806c560 msgr2=0x7f329806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f329806c560 0x7f329806ea20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f32a00015e0 tx=0x7f32a0009040 comp rx=0 tx=0).stop 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 msgr2=0x7f32ac161800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 0x7f32ac161800 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f32a4007e70 tx=0x7f32a400e4b0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 shutdown_connections 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f329806c560 0x7f329806ea20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac15e960 0x7f32ac161800 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32ac070480 0x7f32ac15f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 --2- 172.21.15.156:0/2273146146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32ac06b5b0 0x7f32ac14bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 >> 172.21.15.156:0/2273146146 conn(0x7f32ac072ce0 msgr2=0x7f32ac0af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 shutdown_connections 2024-03-29T04:10:53.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.753+0000 7f32b116d640 1 -- 172.21.15.156:0/2273146146 wait complete. 2024-03-29T04:10:53.744 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T04:10:53.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.872+0000 7fdd94670640 1 Processor -- start 2024-03-29T04:10:53.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.872+0000 7fdd94670640 1 -- start start 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd94670640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd8c15e920 0x7fdd8c0d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd94670640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c0d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd94670640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c0cf190 con 0x7fdd8c0d2120 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd94670640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdd8c0cf7a0 con 0x7fdd8c15e920 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd94670640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d3080 con 0x7fdd8c0cfc30 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd9240c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd8c15e920 0x7fdd8c0d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd91c0b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fdd8c0d2120 0x7fdd8c06c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39752/0 (socket says 172.21.15.156:39752) 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd91c0b640 1 -- 172.21.15.156:0/2756620698 learned_addr learned my addr 172.21.15.156:0/2756620698 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd92c0d640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c0d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd9240c640 1 -- 172.21.15.156:0/2756620698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 msgr2=0x7fdd8c0d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd9240c640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c0d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd9240c640 1 -- 172.21.15.156:0/2756620698 >> v1:172.21.15.156:6789/0 conn(0x7fdd8c0d2120 legacy=0x7fdd8c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd9240c640 1 -- 172.21.15.156:0/2756620698 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c0cf430 con 0x7fdd8c15e920 2024-03-29T04:10:53.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd9140a640 1 -- 172.21.15.156:0/2756620698 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838425314 0 0) 0x7fdd8c0cf190 con 0x7fdd8c0d2120 2024-03-29T04:10:53.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.873+0000 7fdd92c0d640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c0d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:53.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.874+0000 7fdd9240c640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0d26e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009d90 tx=0x7fdd8802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c096e464e280c896 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.874+0000 7fdd9140a640 1 -- 172.21.15.156:0/2756620698 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88039070 con 0x7fdd8c15e920 2024-03-29T04:10:53.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.874+0000 7fdd9140a640 1 -- 172.21.15.156:0/2756620698 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8802cc50 con 0x7fdd8c15e920 2024-03-29T04:10:53.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.874+0000 7fdd9140a640 1 -- 172.21.15.156:0/2756620698 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88034440 con 0x7fdd8c15e920 2024-03-29T04:10:53.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 -- 172.21.15.156:0/2756620698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 msgr2=0x7fdd8c0d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0d26e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009d90 tx=0x7fdd8802bea0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 -- 172.21.15.156:0/2756620698 shutdown_connections 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c0d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 --2- 172.21.15.156:0/2756620698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0d26e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 -- 172.21.15.156:0/2756620698 >> 172.21.15.156:0/2756620698 conn(0x7fdd8c09d7a0 msgr2=0x7fdd8c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 -- 172.21.15.156:0/2756620698 shutdown_connections 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.875+0000 7fdd94670640 1 -- 172.21.15.156:0/2756620698 wait complete. 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.876+0000 7fdd94670640 1 Processor -- start 2024-03-29T04:10:53.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.876+0000 7fdd94670640 1 -- start start 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd94670640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c14d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd94670640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd8c0d2120 0x7fdd8c0cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd94670640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd94670640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d34e0 con 0x7fdd8c15e920 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd94670640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d3360 con 0x7fdd8c0d2120 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd94670640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d2c20 con 0x7fdd8c0cfc30 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38168/0 (socket says 172.21.15.156:38168) 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 -- 172.21.15.156:0/270012069 learned_addr learned my addr 172.21.15.156:0/270012069 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd9240c640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c14d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd91c0b640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd8c0d2120 0x7fdd8c0cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 -- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 msgr2=0x7fdd8c14d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c14d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 -- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd8c0d2120 msgr2=0x7fdd8c0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd8c0d2120 0x7fdd8c0cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd92c0d640 1 -- 172.21.15.156:0/270012069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c0cc130 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.877+0000 7fdd9240c640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c14d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.878+0000 7fdd92c0d640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0cb6b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400c7f0 tx=0x7fdd8400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.878+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84015070 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.878+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd84007820 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.878+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8c0cc410 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.878+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84007c50 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.878+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8c0cc9d0 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.879+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdd84018070 con 0x7fdd8c15e920 2024-03-29T04:10:53.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.879+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8c0a2480 con 0x7fdd8c15e920 2024-03-29T04:10:53.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.879+0000 7fdd82ffd640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd7006c530 0x7fdd7006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:53.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.880+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fdd8408b880 con 0x7fdd8c15e920 2024-03-29T04:10:53.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.880+0000 7fdd9240c640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd7006c530 0x7fdd7006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:53.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.880+0000 7fdd9240c640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd7006c530 0x7fdd7006e9f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdd88036000 tx=0x7fdd88030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:53.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:53.884+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdd84059ea0 con 0x7fdd8c15e920 2024-03-29T04:10:54.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.060+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdd8c04f9d0 con 0x7fdd8c15e920 2024-03-29T04:10:54.040 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.061+0000 7fdd82ffd640 1 -- 172.21.15.156:0/270012069 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7fdd840597a0 con 0x7fdd8c15e920 2024-03-29T04:10:54.041 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 32 2024-03-29T04:10:54.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd7006c530 msgr2=0x7fdd7006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd7006c530 0x7fdd7006e9f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdd88036000 tx=0x7fdd88030000 comp rx=0 tx=0).stop 2024-03-29T04:10:54.042 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 msgr2=0x7fdd8c0cb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0cb6b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400c7f0 tx=0x7fdd8400ccb0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 shutdown_connections 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fdd7006c530 0x7fdd7006e9f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c15e920 0x7fdd8c0cb6b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd8c0d2120 0x7fdd8c0cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 --2- 172.21.15.156:0/270012069 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd8c0cfc30 0x7fdd8c14d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 >> 172.21.15.156:0/270012069 conn(0x7fdd8c09d7a0 msgr2=0x7fdd8c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 shutdown_connections 2024-03-29T04:10:54.043 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.063+0000 7fdd94670640 1 -- 172.21.15.156:0/270012069 wait complete. 2024-03-29T04:10:54.054 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T04:10:54.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.182+0000 7f1da9088640 1 Processor -- start 2024-03-29T04:10:54.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.182+0000 7f1da9088640 1 -- start start 2024-03-29T04:10:54.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da9088640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1da40b59d0 0x7f1da40ae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da9088640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da40b42c0 con 0x7f1da406c5c0 2024-03-29T04:10:54.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da9088640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da40b45a0 con 0x7f1da415e540 2024-03-29T04:10:54.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da9088640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1da4070680 con 0x7f1da40b59d0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da259c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1da40b59d0 0x7f1da40ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da259c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1da40b59d0 0x7f1da40ae850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38180/0 (socket says 172.21.15.156:38180) 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da259c640 1 -- 172.21.15.156:0/1245855440 learned_addr learned my addr 172.21.15.156:0/1245855440 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da359e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1da406c5c0 0x7f1da406e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39768/0 (socket says 172.21.15.156:39768) 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da259c640 1 -- 172.21.15.156:0/1245855440 >> v1:172.21.15.203:6789/0 conn(0x7f1da415e540 legacy=0x7f1da40b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da259c640 1 -- 172.21.15.156:0/1245855440 >> v1:172.21.15.156:6789/0 conn(0x7f1da406c5c0 legacy=0x7f1da406e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da259c640 1 -- 172.21.15.156:0/1245855440 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da40b4880 con 0x7f1da40b59d0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.183+0000 7f1da1d9b640 1 -- 172.21.15.156:0/1245855440 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752626495 0 0) 0x7f1da40b42c0 con 0x7f1da406c5c0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.184+0000 7f1da259c640 1 --2- 172.21.15.156:0/1245855440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da40b59d0 0x7f1da40ae850 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1da4070800 tx=0x7f1d9402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adeaacb0dc373dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.184+0000 7f1da1d9b640 1 -- 172.21.15.156:0/1245855440 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9402f840 con 0x7f1da40b59d0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.184+0000 7f1da1d9b640 1 -- 172.21.15.156:0/1245855440 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9402f9e0 con 0x7f1da40b59d0 2024-03-29T04:10:54.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.184+0000 7f1da1d9b640 1 -- 172.21.15.156:0/1245855440 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9402fdb0 con 0x7f1da40b59d0 2024-03-29T04:10:54.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 -- 172.21.15.156:0/1245855440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da40b59d0 msgr2=0x7f1da40ae850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 --2- 172.21.15.156:0/1245855440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da40b59d0 0x7f1da40ae850 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1da4070800 tx=0x7f1d9402ba20 comp rx=0 tx=0).stop 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 -- 172.21.15.156:0/1245855440 shutdown_connections 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 --2- 172.21.15.156:0/1245855440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da40b59d0 0x7f1da40ae850 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 -- 172.21.15.156:0/1245855440 >> 172.21.15.156:0/1245855440 conn(0x7f1da4072110 msgr2=0x7f1da40724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 -- 172.21.15.156:0/1245855440 shutdown_connections 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.185+0000 7f1da9088640 1 -- 172.21.15.156:0/1245855440 wait complete. 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.186+0000 7f1da9088640 1 Processor -- start 2024-03-29T04:10:54.166 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.186+0000 7f1da9088640 1 -- start start 2024-03-29T04:10:54.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da9088640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1da406c5c0 0x7f1da4145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.168 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da9088640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1da40b59d0 0x7f1da40ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da9088640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 0x7f1da40af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da9088640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1da415e920 con 0x7f1da415e540 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da9088640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40aed90 con 0x7f1da40b59d0 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da9088640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1da415eaa0 con 0x7f1da406c5c0 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 0x7f1da40af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 0x7f1da40af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38194/0 (socket says 172.21.15.156:38194) 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 -- 172.21.15.156:0/3518489279 learned_addr learned my addr 172.21.15.156:0/3518489279 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da2d9d640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1da406c5c0 0x7f1da4145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da259c640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1da40b59d0 0x7f1da40ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 -- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1da406c5c0 msgr2=0x7f1da4145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1da406c5c0 0x7f1da4145630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 -- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1da40b59d0 msgr2=0x7f1da40ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.169 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1da40b59d0 0x7f1da40ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da359e640 1 -- 172.21.15.156:0/3518489279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da41452f0 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da2d9d640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1da406c5c0 0x7f1da4145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.187+0000 7f1da259c640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1da40b59d0 0x7f1da40ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.188+0000 7f1da359e640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 0x7f1da40af760 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00c7d0 tx=0x7f1d9c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.188+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c018070 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.188+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c00a640 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.188+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da41616c0 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.188+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00aa70 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.188+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da4156a60 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.189+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da406e500 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.189+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1d9c00ac10 con 0x7f1da415e540 2024-03-29T04:10:54.170 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.190+0000 7f1d8b7fe640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d9006c560 0x7f1d9006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.190+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f1d9c08acf0 con 0x7f1da415e540 2024-03-29T04:10:54.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.190+0000 7f1da2d9d640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d9006c560 0x7f1d9006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.171 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.190+0000 7f1da2d9d640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d9006c560 0x7f1d9006ea20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d98007fd0 tx=0x7f1d98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.173 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.194+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1da406e500 con 0x7f1da415e540 2024-03-29T04:10:54.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.369+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1da4073740 con 0x7f1da415e540 2024-03-29T04:10:54.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.370+0000 7f1d8b7fe640 1 -- 172.21.15.156:0/3518489279 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7f1d9c08f070 con 0x7f1da415e540 2024-03-29T04:10:54.349 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 64 2024-03-29T04:10:54.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d9006c560 msgr2=0x7f1d9006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d9006c560 0x7f1d9006ea20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d98007fd0 tx=0x7f1d98009040 comp rx=0 tx=0).stop 2024-03-29T04:10:54.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 msgr2=0x7f1da40af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 0x7f1da40af760 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00c7d0 tx=0x7f1d9c00cc90 comp rx=0 tx=0).stop 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 shutdown_connections 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1d9006c560 0x7f1d9006ea20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1da415e540 0x7f1da40af760 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1da40b59d0 0x7f1da40ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 --2- 172.21.15.156:0/3518489279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1da406c5c0 0x7f1da4145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 >> 172.21.15.156:0/3518489279 conn(0x7f1da4072110 msgr2=0x7f1da406f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 shutdown_connections 2024-03-29T04:10:54.352 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.372+0000 7f1da9088640 1 -- 172.21.15.156:0/3518489279 wait complete. 2024-03-29T04:10:54.362 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:10:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.497+0000 7fc9eca74640 1 Processor -- start 2024-03-29T04:10:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.497+0000 7fc9eca74640 1 -- start start 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.498+0000 7fc9eca74640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc9e80d42c0 0x7fc9e80d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.498+0000 7fc9eca74640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e806cbb0 0x7fc9e806d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.498+0000 7fc9eca74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e80d4c80 con 0x7fc9e815e730 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.498+0000 7fc9eca74640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e806c720 con 0x7fc9e80d42c0 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.498+0000 7fc9eca74640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e80cfa20 con 0x7fc9e806cbb0 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.498+0000 7fc9e5d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc9e80d42c0 0x7fc9e80d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e5d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc9e80d42c0 0x7fc9e80d4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38210/0 (socket says 172.21.15.156:38210) 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e5d9b640 1 -- 172.21.15.156:0/112818466 learned_addr learned my addr 172.21.15.156:0/112818466 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e6d9d640 1 --2- 172.21.15.156:0/112818466 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e806cbb0 0x7fc9e806d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e5d9b640 1 -- 172.21.15.156:0/112818466 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e806cbb0 msgr2=0x7fc9e806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e5d9b640 1 --2- 172.21.15.156:0/112818466 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e806cbb0 0x7fc9e806d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e5d9b640 1 -- 172.21.15.156:0/112818466 >> v1:172.21.15.203:6789/0 conn(0x7fc9e815e730 legacy=0x7fc9e80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.499+0000 7fc9e5d9b640 1 -- 172.21.15.156:0/112818466 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e80cfd00 con 0x7fc9e80d42c0 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.500+0000 7fc9e5d9b640 1 --2- 172.21.15.156:0/112818466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e80d42c0 0x7fc9e80d4740 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009850 tx=0x7fc9d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8bc386e3776bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.500+0000 7fc9e559a640 1 -- 172.21.15.156:0/112818466 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0004230 con 0x7fc9e80d42c0 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.500+0000 7fc9e559a640 1 -- 172.21.15.156:0/112818466 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d00043d0 con 0x7fc9e80d42c0 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.500+0000 7fc9e559a640 1 -- 172.21.15.156:0/112818466 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0031830 con 0x7fc9e80d42c0 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.500+0000 7fc9eca74640 1 -- 172.21.15.156:0/112818466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e80d42c0 msgr2=0x7fc9e80d4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.500+0000 7fc9eca74640 1 --2- 172.21.15.156:0/112818466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e80d42c0 0x7fc9e80d4740 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009850 tx=0x7fc9d002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.501+0000 7fc9eca74640 1 -- 172.21.15.156:0/112818466 shutdown_connections 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.501+0000 7fc9eca74640 1 --2- 172.21.15.156:0/112818466 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e806cbb0 0x7fc9e806d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.501+0000 7fc9eca74640 1 --2- 172.21.15.156:0/112818466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e80d42c0 0x7fc9e80d4740 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.501+0000 7fc9eca74640 1 -- 172.21.15.156:0/112818466 >> 172.21.15.156:0/112818466 conn(0x7fc9e808bfd0 msgr2=0x7fc9e809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:54.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.501+0000 7fc9eca74640 1 -- 172.21.15.156:0/112818466 shutdown_connections 2024-03-29T04:10:54.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.501+0000 7fc9eca74640 1 -- 172.21.15.156:0/112818466 wait complete. 2024-03-29T04:10:54.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.503+0000 7fc9eca74640 1 Processor -- start 2024-03-29T04:10:54.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.503+0000 7fc9eca74640 1 -- start start 2024-03-29T04:10:54.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9eca74640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 0x7fc9e8160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9eca74640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e80d42c0 0x7fc9e8161810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9eca74640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e815e730 0x7fc9e814aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9eca74640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e80d2220 con 0x7fc9e806cbb0 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9eca74640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e80d02c0 con 0x7fc9e815e730 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9eca74640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e815eb10 con 0x7fc9e80d42c0 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 0x7fc9e8160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 0x7fc9e8160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38212/0 (socket says 172.21.15.156:38212) 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 -- 172.21.15.156:0/731836280 learned_addr learned my addr 172.21.15.156:0/731836280 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e6d9d640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e815e730 0x7fc9e814aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e5d9b640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e80d42c0 0x7fc9e8161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 -- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e80d42c0 msgr2=0x7fc9e8161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e80d42c0 0x7fc9e8161810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 -- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e815e730 msgr2=0x7fc9e814aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e815e730 0x7fc9e814aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 -- 172.21.15.156:0/731836280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e81609b0 con 0x7fc9e806cbb0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e5d9b640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e80d42c0 0x7fc9e8161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.504+0000 7fc9e659c640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 0x7fc9e8160340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000ba80 tx=0x7fc9e000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.505+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000cc00 con 0x7fc9e806cbb0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.505+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e000cda0 con 0x7fc9e806cbb0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.505+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e8160c90 con 0x7fc9e806cbb0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.505+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0003bb0 con 0x7fc9e806cbb0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.505+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e80cc680 con 0x7fc9e806cbb0 2024-03-29T04:10:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.506+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e8068610 con 0x7fc9e806cbb0 2024-03-29T04:10:54.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.506+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc9e0014440 con 0x7fc9e806cbb0 2024-03-29T04:10:54.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.507+0000 7fc9deffd640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc9c806c530 0x7fc9c806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.507+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc9e008b5d0 con 0x7fc9e806cbb0 2024-03-29T04:10:54.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.507+0000 7fc9e5d9b640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc9c806c530 0x7fc9c806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.507+0000 7fc9e5d9b640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc9c806c530 0x7fc9c806e9f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0035000 tx=0x7fc9d0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.511+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc9e0059c80 con 0x7fc9e806cbb0 2024-03-29T04:10:54.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.726+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9e80d0540 con 0x7fc9e806cbb0 2024-03-29T04:10:54.706 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.727+0000 7fc9deffd640 1 -- 172.21.15.156:0/731836280 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5190 (secure 0 0 0) 0x7fc9e0059580 con 0x7fc9e806cbb0 2024-03-29T04:10:54.706 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:54.706 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:09:53.360769+0000","modified":"2024-03-29T04:10:10.696817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5310":{"gid":5310,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/788767406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":788767406},{"type":"v1","addr":"172.21.15.156:6837","nonce":788767406}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5310,"qdb_cluster":[5310]},"id":2}]} 2024-03-29T04:10:54.707 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 17 2024-03-29T04:10:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc9c806c530 msgr2=0x7fc9c806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc9c806c530 0x7fc9c806e9f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0035000 tx=0x7fc9d0008030 comp rx=0 tx=0).stop 2024-03-29T04:10:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 msgr2=0x7fc9e8160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 0x7fc9e8160340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000ba80 tx=0x7fc9e000bf40 comp rx=0 tx=0).stop 2024-03-29T04:10:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 shutdown_connections 2024-03-29T04:10:54.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc9c806c530 0x7fc9c806e9f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e815e730 0x7fc9e814aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc9e80d42c0 0x7fc9e8161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 --2- 172.21.15.156:0/731836280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e806cbb0 0x7fc9e8160340 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 >> 172.21.15.156:0/731836280 conn(0x7fc9e808bfd0 msgr2=0x7fc9e80d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:54.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 shutdown_connections 2024-03-29T04:10:54.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.729+0000 7fc9eca74640 1 -- 172.21.15.156:0/731836280 wait complete. 2024-03-29T04:10:54.719 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:10:54.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.842+0000 7f9241dcd640 1 Processor -- start 2024-03-29T04:10:54.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.842+0000 7f9241dcd640 1 -- start start 2024-03-29T04:10:54.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.843+0000 7f9241dcd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f923c06dbc0 0x7f923c169350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.843+0000 7f9241dcd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f923c0cfa20 con 0x7f923c0d42c0 2024-03-29T04:10:54.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.843+0000 7f9241dcd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f923c0cfd00 con 0x7f923c15e730 2024-03-29T04:10:54.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.843+0000 7f9241dcd640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f923c06c720 con 0x7f923c06dbc0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.843+0000 7f923affd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f923c0d42c0 0x7f923c0d4740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39780/0 (socket says 172.21.15.156:39780) 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.843+0000 7f923affd640 1 -- 172.21.15.156:0/227735463 learned_addr learned my addr 172.21.15.156:0/227735463 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923bfff640 1 --2- 172.21.15.156:0/227735463 >> v2:172.21.15.203:3300/0 conn(0x7f923c06dbc0 0x7f923c169350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836735253 0 0) 0x7f923c0cfa20 con 0x7f923c0d42c0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9224003520 con 0x7f923c0d42c0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4065507363 0 0) 0x7f9224003520 con 0x7f923c0d42c0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 >> v2:172.21.15.203:3300/0 conn(0x7f923c06dbc0 msgr2=0x7f923c169350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 --2- 172.21.15.156:0/227735463 >> v2:172.21.15.203:3300/0 conn(0x7f923c06dbc0 0x7f923c169350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 >> v1:172.21.15.203:6789/0 conn(0x7f923c15e730 legacy=0x7f923c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c0cffe0 con 0x7f923c0d42c0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f922c0016c0 con 0x7f923c0d42c0 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 >> v1:172.21.15.156:6789/0 conn(0x7f923c0d42c0 legacy=0x7f923c0d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92240039b0 0x7f9224003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9224004800 0x7f9224004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92240051c0 0x7f922400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f923c06c720 con 0x7f92240039b0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f922400c060 con 0x7f92240051c0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f922400c260 con 0x7f9224004800 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923b7fe640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92240039b0 0x7f9224003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923bfff640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92240051c0 0x7f922400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923affd640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9224004800 0x7f9224004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923b7fe640 1 -- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9224004800 msgr2=0x7f9224004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923b7fe640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9224004800 0x7f9224004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923b7fe640 1 -- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92240051c0 msgr2=0x7f922400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923b7fe640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92240051c0 0x7f922400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923b7fe640 1 -- 172.21.15.156:0/227735463 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9224003520 con 0x7f92240039b0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.844+0000 7f923bfff640 1 -- 172.21.15.156:0/227735463 reap_dead start 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.845+0000 7f923b7fe640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92240039b0 0x7f9224003dd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9230001750 tx=0x7f9224005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749e88ded5df6d54 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.845+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230001370 con 0x7f92240039b0 2024-03-29T04:10:54.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.845+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92300052d0 con 0x7f92240039b0 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.845+0000 7f923a7fc640 1 -- 172.21.15.156:0/227735463 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230003830 con 0x7f92240039b0 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 -- 172.21.15.156:0/227735463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92240039b0 msgr2=0x7f9224003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92240039b0 0x7f9224003dd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9230001750 tx=0x7f9224005830 comp rx=0 tx=0).stop 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 -- 172.21.15.156:0/227735463 shutdown_connections 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 --2- 172.21.15.156:0/227735463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92240039b0 0x7f9224003dd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 -- 172.21.15.156:0/227735463 >> 172.21.15.156:0/227735463 conn(0x7f923c08bfd0 msgr2=0x7f923c09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 -- 172.21.15.156:0/227735463 shutdown_connections 2024-03-29T04:10:54.828 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.846+0000 7f9241dcd640 1 -- 172.21.15.156:0/227735463 wait complete. 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.848+0000 7f9241dcd640 1 Processor -- start 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.848+0000 7f9241dcd640 1 -- start start 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f9241dcd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92240039b0 0x7f923c14eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f9241dcd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f923c0b4790 0x7f923c0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f9241dcd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 0x7f923c160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f9241dcd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f923c15eb10 con 0x7f923c15fee0 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f9241dcd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f923c0d2220 con 0x7f923c0b4790 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f9241dcd640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f923c15ec90 con 0x7f92240039b0 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f923bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 0x7f923c160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f923bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 0x7f923c160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38238/0 (socket says 172.21.15.156:38238) 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.849+0000 7f923bfff640 1 -- 172.21.15.156:0/251950211 learned_addr learned my addr 172.21.15.156:0/251950211 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923b7fe640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92240039b0 0x7f923c14eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923bfff640 1 -- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92240039b0 msgr2=0x7f923c14eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923bfff640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92240039b0 0x7f923c14eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923affd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f923c0b4790 0x7f923c0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923bfff640 1 -- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f923c0b4790 msgr2=0x7f923c0b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923bfff640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f923c0b4790 0x7f923c0b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923bfff640 1 -- 172.21.15.156:0/251950211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c0c9630 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f923bfff640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 0x7f923c160360 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9234011b40 tx=0x7f923400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923400fd10 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9234015430 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234015820 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923c0c9910 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.850+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f923c0c9ed0 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.851+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f923c068610 con 0x7f923c15fee0 2024-03-29T04:10:54.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.852+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f92340159c0 con 0x7f923c15fee0 2024-03-29T04:10:54.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.852+0000 7f9223fff640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f922806c560 0x7f922806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:54.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.852+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f923408c430 con 0x7f923c15fee0 2024-03-29T04:10:54.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.852+0000 7f923b7fe640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f922806c560 0x7f922806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:54.833 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.853+0000 7f923b7fe640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f922806c560 0x7f922806ea20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9230033000 tx=0x7f9230009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:54.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:54.861+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9234019030 con 0x7f923c15fee0 2024-03-29T04:10:55.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.042+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f923c15dea0 con 0x7f923c15fee0 2024-03-29T04:10:55.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.044+0000 7f9223fff640 1 -- 172.21.15.156:0/251950211 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7f9234010040 con 0x7f923c15fee0 2024-03-29T04:10:55.023 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:55.023 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":34,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:55.015332+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:10:55.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f922806c560 msgr2=0x7f922806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f922806c560 0x7f922806ea20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9230033000 tx=0x7f9230009030 comp rx=0 tx=0).stop 2024-03-29T04:10:55.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 msgr2=0x7f923c160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 0x7f923c160360 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9234011b40 tx=0x7f923400f1b0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 shutdown_connections 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f922806c560 0x7f922806ea20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f923c15fee0 0x7f923c160360 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f923c0b4790 0x7f923c0b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 --2- 172.21.15.156:0/251950211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92240039b0 0x7f923c14eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 >> 172.21.15.156:0/251950211 conn(0x7f923c08bfd0 msgr2=0x7f923c0b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 shutdown_connections 2024-03-29T04:10:55.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.046+0000 7f9241dcd640 1 -- 172.21.15.156:0/251950211 wait complete. 2024-03-29T04:10:55.038 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T04:10:55.038 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:10:55.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.173+0000 7f7e17104640 1 Processor -- start 2024-03-29T04:10:55.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.173+0000 7f7e17104640 1 -- start start 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e17104640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7e100af6b0 0x7f7e1015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e17104640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e1006f7a0 0x7f7e10071750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e17104640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1015c7e0 con 0x7f7e100b3050 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e17104640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7e100afa90 con 0x7f7e1006f7a0 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e17104640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7e100b3510 con 0x7f7e100af6b0 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e156a1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e1006f7a0 0x7f7e10071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e0ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7e100b3050 0x7f7e1006f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39796/0 (socket says 172.21.15.156:39796) 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e156a1640 1 --2- 172.21.15.156:0/4294217491 >> v2:172.21.15.156:3300/0 conn(0x7f7e1006f7a0 0x7f7e10071750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38248/0 (socket says 172.21.15.156:38248) 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.174+0000 7f7e0ffff640 1 -- 172.21.15.156:0/4294217491 learned_addr learned my addr 172.21.15.156:0/4294217491 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e14ea0640 1 --2- 172.21.15.156:0/4294217491 >> v2:172.21.15.203:3300/0 conn(0x7f7e100af6b0 0x7f7e1015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e156a1640 1 -- 172.21.15.156:0/4294217491 >> v2:172.21.15.203:3300/0 conn(0x7f7e100af6b0 msgr2=0x7f7e1015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e156a1640 1 --2- 172.21.15.156:0/4294217491 >> v2:172.21.15.203:3300/0 conn(0x7f7e100af6b0 0x7f7e1015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e156a1640 1 -- 172.21.15.156:0/4294217491 >> v1:172.21.15.156:6789/0 conn(0x7f7e100b3050 legacy=0x7f7e1006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e0f7fe640 1 -- 172.21.15.156:0/4294217491 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224152911 0 0) 0x7f7e00001110 con 0x7f7e100b3050 2024-03-29T04:10:55.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e156a1640 1 -- 172.21.15.156:0/4294217491 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e100b37f0 con 0x7f7e1006f7a0 2024-03-29T04:10:55.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.175+0000 7f7e156a1640 1 --2- 172.21.15.156:0/4294217491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10071750 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7e1006fc20 tx=0x7f7df802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220cb03fd5dc1714 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e0f7fe640 1 -- 172.21.15.156:0/4294217491 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df80315a0 con 0x7f7e1006f7a0 2024-03-29T04:10:55.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e0f7fe640 1 -- 172.21.15.156:0/4294217491 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df8037070 con 0x7f7e1006f7a0 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e0f7fe640 1 -- 172.21.15.156:0/4294217491 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8031960 con 0x7f7e1006f7a0 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 -- 172.21.15.156:0/4294217491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 msgr2=0x7f7e10071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 --2- 172.21.15.156:0/4294217491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10071750 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7e1006fc20 tx=0x7f7df802b840 comp rx=0 tx=0).stop 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 -- 172.21.15.156:0/4294217491 shutdown_connections 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 --2- 172.21.15.156:0/4294217491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10071750 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 --2- 172.21.15.156:0/4294217491 >> v2:172.21.15.203:3300/0 conn(0x7f7e100af6b0 0x7f7e1015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 -- 172.21.15.156:0/4294217491 >> 172.21.15.156:0/4294217491 conn(0x7f7e100b1d50 msgr2=0x7f7e100b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 -- 172.21.15.156:0/4294217491 shutdown_connections 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.176+0000 7f7e17104640 1 -- 172.21.15.156:0/4294217491 wait complete. 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.177+0000 7f7e17104640 1 Processor -- start 2024-03-29T04:10:55.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.177+0000 7f7e17104640 1 -- start start 2024-03-29T04:10:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e17104640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e17104640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e100af6b0 0x7f7e100c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e17104640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e100b3050 0x7f7e100c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e17104640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e100b4090 con 0x7f7e1006f7a0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e17104640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e100b3db0 con 0x7f7e100b3050 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e17104640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7e100b45b0 con 0x7f7e100af6b0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e14ea0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e14ea0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10149160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38254/0 (socket says 172.21.15.156:38254) 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.178+0000 7f7e14ea0640 1 -- 172.21.15.156:0/1707976066 learned_addr learned my addr 172.21.15.156:0/1707976066 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e0ffff640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e100af6b0 0x7f7e100c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e14ea0640 1 -- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e100af6b0 msgr2=0x7f7e100c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e14ea0640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e100af6b0 0x7f7e100c8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e14ea0640 1 -- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e100b3050 msgr2=0x7f7e100c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e14ea0640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e100b3050 0x7f7e100c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e14ea0640 1 -- 172.21.15.156:0/1707976066 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e100ca4a0 con 0x7f7e1006f7a0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e14ea0640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10149160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7e08007ef0 tx=0x7f7e0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08014620 con 0x7f7e1006f7a0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e08019070 con 0x7f7e1006f7a0 2024-03-29T04:10:55.160 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08014a00 con 0x7f7e1006f7a0 2024-03-29T04:10:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e100ca780 con 0x7f7e1006f7a0 2024-03-29T04:10:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.179+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e100cad40 con 0x7f7e1006f7a0 2024-03-29T04:10:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.180+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e10068610 con 0x7f7e1006f7a0 2024-03-29T04:10:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.180+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7e08014ba0 con 0x7f7e1006f7a0 2024-03-29T04:10:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.181+0000 7f7e0d7fa640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7df006c530 0x7f7df006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.181+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f7e0808c040 con 0x7f7e1006f7a0 2024-03-29T04:10:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.181+0000 7f7e0ffff640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7df006c530 0x7f7df006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.181+0000 7f7e0ffff640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7df006c530 0x7f7df006e9f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e101f56d0 tx=0x7f7e00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.185+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7e0805a660 con 0x7f7e1006f7a0 2024-03-29T04:10:55.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.458+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e1006da70 con 0x7f7e1006f7a0 2024-03-29T04:10:55.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.459+0000 7f7e0d7fa640 1 -- 172.21.15.156:0/1707976066 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5190 (secure 0 0 0) 0x7f7e08059f60 con 0x7f7e1006f7a0 2024-03-29T04:10:55.438 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:55.438 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:09:53.360769+0000","modified":"2024-03-29T04:10:10.696817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5310":{"gid":5310,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/788767406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":788767406},{"type":"v1","addr":"172.21.15.156:6837","nonce":788767406}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5310,"qdb_cluster":[5310]},"id":2}]} 2024-03-29T04:10:55.439 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 17 2024-03-29T04:10:55.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7df006c530 msgr2=0x7f7df006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7df006c530 0x7f7df006e9f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e101f56d0 tx=0x7f7e00009040 comp rx=0 tx=0).stop 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 msgr2=0x7f7e10149160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10149160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7e08007ef0 tx=0x7f7e0800e4b0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 shutdown_connections 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7df006c530 0x7f7df006e9f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e100b3050 0x7f7e100c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e100af6b0 0x7f7e100c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 --2- 172.21.15.156:0/1707976066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e1006f7a0 0x7f7e10149160 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 >> 172.21.15.156:0/1707976066 conn(0x7f7e100b1d50 msgr2=0x7f7e10071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 shutdown_connections 2024-03-29T04:10:55.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.461+0000 7f7e17104640 1 -- 172.21.15.156:0/1707976066 wait complete. 2024-03-29T04:10:55.452 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:10:55.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.574+0000 7f4ac3d32640 1 Processor -- start 2024-03-29T04:10:55.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.574+0000 7f4ac3d32640 1 -- start start 2024-03-29T04:10:55.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac3d32640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4abc15e730 0x7f4abc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac3d32640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4abc0d49d0 0x7f4abc06cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac3d32640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4abc0cfa20 con 0x7f4abc0cf3e0 2024-03-29T04:10:55.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac3d32640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f4abc06c720 con 0x7f4abc0d49d0 2024-03-29T04:10:55.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac3d32640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4abc0cfd00 con 0x7f4abc15e730 2024-03-29T04:10:55.555 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac22cf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4abc0d49d0 0x7f4abc06cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac22cf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4abc0d49d0 0x7f4abc06cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38258/0 (socket says 172.21.15.156:38258) 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac12cd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4abc0cf3e0 0x7f4abc0d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39798/0 (socket says 172.21.15.156:39798) 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac22cf640 1 -- 172.21.15.156:0/1941520343 learned_addr learned my addr 172.21.15.156:0/1941520343 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.575+0000 7f4ac1ace640 1 --2- 172.21.15.156:0/1941520343 >> v2:172.21.15.203:3300/0 conn(0x7f4abc15e730 0x7f4abc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.576+0000 7f4ac22cf640 1 -- 172.21.15.156:0/1941520343 >> v2:172.21.15.203:3300/0 conn(0x7f4abc15e730 msgr2=0x7f4abc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.576+0000 7f4ac22cf640 1 --2- 172.21.15.156:0/1941520343 >> v2:172.21.15.203:3300/0 conn(0x7f4abc15e730 0x7f4abc0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.576+0000 7f4ac22cf640 1 -- 172.21.15.156:0/1941520343 >> v1:172.21.15.156:6789/0 conn(0x7f4abc0cf3e0 legacy=0x7f4abc0d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.576+0000 7f4ac0acc640 1 -- 172.21.15.156:0/1941520343 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575848900 0 0) 0x7f4abc0cfa20 con 0x7f4abc0cf3e0 2024-03-29T04:10:55.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.576+0000 7f4ac22cf640 1 -- 172.21.15.156:0/1941520343 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4abc0cffe0 con 0x7f4abc0d49d0 2024-03-29T04:10:55.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.576+0000 7f4ac22cf640 1 --2- 172.21.15.156:0/1941520343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc0d49d0 0x7f4abc06cbb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0d1f20 tx=0x7f4ab402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafcec539a0e6b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac0acc640 1 -- 172.21.15.156:0/1941520343 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab402ca80 con 0x7f4abc0d49d0 2024-03-29T04:10:55.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac0acc640 1 -- 172.21.15.156:0/1941520343 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab402cc20 con 0x7f4abc0d49d0 2024-03-29T04:10:55.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac0acc640 1 -- 172.21.15.156:0/1941520343 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40343f0 con 0x7f4abc0d49d0 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 -- 172.21.15.156:0/1941520343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc0d49d0 msgr2=0x7f4abc06cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/1941520343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc0d49d0 0x7f4abc06cbb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0d1f20 tx=0x7f4ab402bee0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 -- 172.21.15.156:0/1941520343 shutdown_connections 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/1941520343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc0d49d0 0x7f4abc06cbb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/1941520343 >> v2:172.21.15.203:3300/0 conn(0x7f4abc15e730 0x7f4abc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 -- 172.21.15.156:0/1941520343 >> 172.21.15.156:0/1941520343 conn(0x7f4abc08bfd0 msgr2=0x7f4abc09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 -- 172.21.15.156:0/1941520343 shutdown_connections 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.577+0000 7f4ac3d32640 1 -- 172.21.15.156:0/1941520343 wait complete. 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.578+0000 7f4ac3d32640 1 Processor -- start 2024-03-29T04:10:55.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.578+0000 7f4ac3d32640 1 -- start start 2024-03-29T04:10:55.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac3d32640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0cf3e0 0x7f4abc0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac3d32640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4abc0d49d0 0x7f4abc153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac3d32640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 0x7f4abc156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac3d32640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4abc15eb10 con 0x7f4abc15e730 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac3d32640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4abc0d2220 con 0x7f4abc0cf3e0 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac3d32640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4abc15ec90 con 0x7f4abc0d49d0 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac22cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 0x7f4abc156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac22cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 0x7f4abc156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38272/0 (socket says 172.21.15.156:38272) 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.579+0000 7f4ac22cf640 1 -- 172.21.15.156:0/2823862050 learned_addr learned my addr 172.21.15.156:0/2823862050 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac1ace640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0cf3e0 0x7f4abc0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac12cd640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4abc0d49d0 0x7f4abc153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac22cf640 1 -- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4abc0d49d0 msgr2=0x7f4abc153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac22cf640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4abc0d49d0 0x7f4abc153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac22cf640 1 -- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0cf3e0 msgr2=0x7f4abc0c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac22cf640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0cf3e0 0x7f4abc0c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac22cf640 1 -- 172.21.15.156:0/2823862050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4abc0c95d0 con 0x7f4abc15e730 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac1ace640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0cf3e0 0x7f4abc0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac22cf640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 0x7f4abc156c50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009f40 tx=0x7f4ab4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4043030 con 0x7f4abc15e730 2024-03-29T04:10:55.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab4003990 con 0x7f4abc15e730 2024-03-29T04:10:55.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4abc0c98b0 con 0x7f4abc15e730 2024-03-29T04:10:55.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4009510 con 0x7f4abc15e730 2024-03-29T04:10:55.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.580+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4abc0ca150 con 0x7f4abc15e730 2024-03-29T04:10:55.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.581+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4abc0c9a80 con 0x7f4abc15e730 2024-03-29T04:10:55.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.581+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4ab402ca90 con 0x7f4abc15e730 2024-03-29T04:10:55.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.582+0000 7f4ab27fc640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4aa006c530 0x7f4aa006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.582+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4ab40adc50 con 0x7f4abc15e730 2024-03-29T04:10:55.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.582+0000 7f4ac1ace640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4aa006c530 0x7f4aa006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.582+0000 7f4ac1ace640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4aa006c530 0x7f4aa006e9f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009a20 tx=0x7f4ab8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.590+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4ab40b3020 con 0x7f4abc15e730 2024-03-29T04:10:55.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.804+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4abc068610 con 0x7f4abc15e730 2024-03-29T04:10:55.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.805+0000 7f4ab27fc640 1 -- 172.21.15.156:0/2823862050 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5190 (secure 0 0 0) 0x7f4ab407c270 con 0x7f4abc15e730 2024-03-29T04:10:55.784 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:55.785 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T04:09:53.360769+0000","modified":"2024-03-29T04:10:10.696817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5310":{"gid":5310,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/788767406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":788767406},{"type":"v1","addr":"172.21.15.156:6837","nonce":788767406}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5310,"qdb_cluster":[5310]},"id":2}]} 2024-03-29T04:10:55.785 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 17 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4aa006c530 msgr2=0x7f4aa006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4aa006c530 0x7f4aa006e9f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009a20 tx=0x7f4ab8008040 comp rx=0 tx=0).stop 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 msgr2=0x7f4abc156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 0x7f4abc156c50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009f40 tx=0x7f4ab4004a40 comp rx=0 tx=0).stop 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 shutdown_connections 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f4aa006c530 0x7f4aa006e9f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4abc15e730 0x7f4abc156c50 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4abc0d49d0 0x7f4abc153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 --2- 172.21.15.156:0/2823862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0cf3e0 0x7f4abc0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 >> 172.21.15.156:0/2823862050 conn(0x7f4abc08bfd0 msgr2=0x7f4abc0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 shutdown_connections 2024-03-29T04:10:55.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.807+0000 7f4ac3d32640 1 -- 172.21.15.156:0/2823862050 wait complete. 2024-03-29T04:10:55.797 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:10:55.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.920+0000 7fa27e332640 1 Processor -- start 2024-03-29T04:10:55.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.920+0000 7fa27e332640 1 -- start start 2024-03-29T04:10:55.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27e332640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 0x7fa2780ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27e332640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa2780cf3e0 0x7fa2780d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27e332640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2780d4c90 con 0x7fa2780d4800 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27e332640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa27806c720 con 0x7fa2780cf3e0 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27e332640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa2780cfa20 con 0x7fa27815e730 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27c8cf640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa2780d4800 0x7fa2780d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39812/0 (socket says 172.21.15.156:39812) 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa27c8cf640 1 -- 172.21.15.156:0/1186213684 learned_addr learned my addr 172.21.15.156:0/1186213684 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa277fff640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 0x7fa2780ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa2777fe640 1 --2- 172.21.15.156:0/1186213684 >> v2:172.21.15.203:3300/0 conn(0x7fa2780cf3e0 0x7fa2780d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590497382 0 0) 0x7fa2780d4c90 con 0x7fa2780d4800 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.921+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa260003520 con 0x7fa2780d4800 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2327412060 0 0) 0x7fa260003520 con 0x7fa2780d4800 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 msgr2=0x7fa2780ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 0x7fa2780ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 >> v2:172.21.15.203:3300/0 conn(0x7fa2780cf3e0 msgr2=0x7fa2780d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 --2- 172.21.15.156:0/1186213684 >> v2:172.21.15.203:3300/0 conn(0x7fa2780cf3e0 0x7fa2780d42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2780cfd00 con 0x7fa2780d4800 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fa2700019f0 con 0x7fa2780d4800 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa2777fe640 1 --2- 172.21.15.156:0/1186213684 >> v2:172.21.15.203:3300/0 conn(0x7fa2780cf3e0 0x7fa2780d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 >> v1:172.21.15.156:6789/0 conn(0x7fa2780d4800 legacy=0x7fa2780d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2600039b0 0x7fa260003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 0x7fa260004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa260004d40 0x7fa2600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa27806c720 con 0x7fa260004d40 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2780cfa20 con 0x7fa2600039b0 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa26000b9c0 con 0x7fa27815e730 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa27c8cf640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa260004d40 0x7fa2600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa2777fe640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 0x7fa260004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa277fff640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2600039b0 0x7fa260003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.905 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa27c8cf640 1 -- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 msgr2=0x7fa260004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa27c8cf640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa27815e730 0x7fa260004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa27c8cf640 1 -- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2600039b0 msgr2=0x7fa260003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa27c8cf640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2600039b0 0x7fa260003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.922+0000 7fa27c8cf640 1 -- 172.21.15.156:0/1186213684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa260003520 con 0x7fa260004d40 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.923+0000 7fa27c8cf640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa260004d40 0x7fa2600051c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa2700013d0 tx=0x7fa260005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a910fef091617da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.923+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27000a4e0 con 0x7fa260004d40 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.923+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa270034650 con 0x7fa260004d40 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.923+0000 7fa276ffd640 1 -- 172.21.15.156:0/1186213684 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa270034a40 con 0x7fa260004d40 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27e332640 1 -- 172.21.15.156:0/1186213684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa260004d40 msgr2=0x7fa2600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27e332640 1 --2- 172.21.15.156:0/1186213684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa260004d40 0x7fa2600051c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa2700013d0 tx=0x7fa260005830 comp rx=0 tx=0).stop 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27c8cf640 1 -- 172.21.15.156:0/1186213684 reap_dead start 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27e332640 1 -- 172.21.15.156:0/1186213684 shutdown_connections 2024-03-29T04:10:55.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27e332640 1 -- 172.21.15.156:0/1186213684 >> 172.21.15.156:0/1186213684 conn(0x7fa27808bfd0 msgr2=0x7fa27809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:55.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27e332640 1 -- 172.21.15.156:0/1186213684 shutdown_connections 2024-03-29T04:10:55.907 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.924+0000 7fa27e332640 1 -- 172.21.15.156:0/1186213684 wait complete. 2024-03-29T04:10:55.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.925+0000 7fa27e332640 1 Processor -- start 2024-03-29T04:10:55.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.925+0000 7fa27e332640 1 -- start start 2024-03-29T04:10:55.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27e332640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2780cb770 0x7fa2780b9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27e332640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 0x7fa2780cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27e332640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa27814e450 0x7fa27814e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27e332640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2780d2220 con 0x7fa2780cbb50 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27e332640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2780d02c0 con 0x7fa27814e450 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27e332640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa27815eb10 con 0x7fa2780cb770 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 0x7fa2780cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 0x7fa2780cdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38300/0 (socket says 172.21.15.156:38300) 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 -- 172.21.15.156:0/729408731 learned_addr learned my addr 172.21.15.156:0/729408731 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27c8cf640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa27814e450 0x7fa27814e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa277fff640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2780cb770 0x7fa2780b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 -- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2780cb770 msgr2=0x7fa2780b9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2780cb770 0x7fa2780b9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 -- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa27814e450 msgr2=0x7fa27814e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa27814e450 0x7fa27814e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 -- 172.21.15.156:0/729408731 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27814ee10 con 0x7fa2780cbb50 2024-03-29T04:10:55.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa27c8cf640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa27814e450 0x7fa27814e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.926+0000 7fa2777fe640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 0x7fa2780cdf50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa268011cd0 tx=0x7fa26800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.927+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26800fd20 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.927+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa268004020 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.927+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2780b8360 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.927+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2680043f0 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.927+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2780b8920 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.928+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa278068610 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.929+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa268012040 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.929+0000 7fa274ff9640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa26406c560 0x7fa26406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.929+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fa26808bce0 con 0x7fa2780cbb50 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.929+0000 7fa277fff640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa26406c560 0x7fa26406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:55.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.929+0000 7fa277fff640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa26406c560 0x7fa26406ea20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009980 tx=0x7fa26c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:55.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:55.933+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa268019030 con 0x7fa2780cbb50 2024-03-29T04:10:56.017 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:10:56.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:10:56.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.113+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa27815df50 con 0x7fa2780cbb50 2024-03-29T04:10:56.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.114+0000 7fa274ff9640 1 -- 172.21.15.156:0/729408731 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7fa26805a300 con 0x7fa2780cbb50 2024-03-29T04:10:56.094 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:10:56.094 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":34,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:09:55.015332+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T04:09:51.822722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T04:09:52.315661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa26406c560 msgr2=0x7fa26406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa26406c560 0x7fa26406ea20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009980 tx=0x7fa26c008040 comp rx=0 tx=0).stop 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 msgr2=0x7fa2780cdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 0x7fa2780cdf50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa268011cd0 tx=0x7fa26800f610 comp rx=0 tx=0).stop 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 shutdown_connections 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa26406c560 0x7fa26406ea20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa27814e450 0x7fa27814e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2780cbb50 0x7fa2780cdf50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 --2- 172.21.15.156:0/729408731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa2780cb770 0x7fa2780b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 >> 172.21.15.156:0/729408731 conn(0x7fa27808bfd0 msgr2=0x7fa27815bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:56.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 shutdown_connections 2024-03-29T04:10:56.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.116+0000 7fa27e332640 1 -- 172.21.15.156:0/729408731 wait complete. 2024-03-29T04:10:56.107 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T04:10:56.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.271+0000 7fb0d5552640 1 Processor -- start 2024-03-29T04:10:56.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.272+0000 7fb0d5552640 1 -- start start 2024-03-29T04:10:56.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.272+0000 7fb0d5552640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d015e920 0x7fb0d00d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.272+0000 7fb0d5552640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb0d00d2120 0x7fb0d006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.272+0000 7fb0d5552640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d00cf190 con 0x7fb0d006cb10 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.272+0000 7fb0d5552640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d00cf7a0 con 0x7fb0d00d2120 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.272+0000 7fb0d5552640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d00d3080 con 0x7fb0d015e920 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ceffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d015e920 0x7fb0d00d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ce7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb0d00d2120 0x7fb0d006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0cf7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb0d006cb10 0x7fb0d00cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:58490/0 (socket says 172.21.15.156:58490) 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0cf7fe640 1 -- 172.21.15.156:0/3320257830 learned_addr learned my addr 172.21.15.156:0/3320257830 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0cdffb640 1 -- 172.21.15.156:0/3320257830 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453241707 0 0) 0x7fb0d00cf190 con 0x7fb0d006cb10 2024-03-29T04:10:56.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0cdffb640 1 -- 172.21.15.156:0/3320257830 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0b8003520 con 0x7fb0d006cb10 2024-03-29T04:10:56.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ce7fc640 1 -- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d015e920 msgr2=0x7fb0d00d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:56.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ce7fc640 1 --2- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d015e920 0x7fb0d00d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ce7fc640 1 -- 172.21.15.156:0/3320257830 >> v1:172.21.15.203:6789/0 conn(0x7fb0d006cb10 legacy=0x7fb0d00cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ce7fc640 1 -- 172.21.15.156:0/3320257830 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d00cf430 con 0x7fb0d00d2120 2024-03-29T04:10:56.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.273+0000 7fb0ceffd640 1 --2- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d015e920 0x7fb0d00d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.274+0000 7fb0ce7fc640 1 --2- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d006c5d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40097a0 tx=0x7fb0c402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7161bf6e4b389dfd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.274+0000 7fb0cdffb640 1 -- 172.21.15.156:0/3320257830 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4035020 con 0x7fb0d00d2120 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.274+0000 7fb0cdffb640 1 -- 172.21.15.156:0/3320257830 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c4031830 con 0x7fb0d00d2120 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.274+0000 7fb0cdffb640 1 -- 172.21.15.156:0/3320257830 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4031be0 con 0x7fb0d00d2120 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 -- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 msgr2=0x7fb0d006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 --2- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d006c5d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40097a0 tx=0x7fb0c402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 -- 172.21.15.156:0/3320257830 shutdown_connections 2024-03-29T04:10:56.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 --2- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d006c5d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 --2- 172.21.15.156:0/3320257830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d015e920 0x7fb0d00d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 -- 172.21.15.156:0/3320257830 >> 172.21.15.156:0/3320257830 conn(0x7fb0d009d7a0 msgr2=0x7fb0d00d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:56.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 -- 172.21.15.156:0/3320257830 shutdown_connections 2024-03-29T04:10:56.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.275+0000 7fb0d5552640 1 -- 172.21.15.156:0/3320257830 wait complete. 2024-03-29T04:10:56.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.277+0000 7fb0d5552640 1 Processor -- start 2024-03-29T04:10:56.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.277+0000 7fb0d5552640 1 -- start start 2024-03-29T04:10:56.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0d5552640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d006cb10 0x7fb0d00b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0d5552640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d01f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0d5552640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 0x7fb0d0152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0d5552640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00d04c0 con 0x7fb0d015e920 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0d5552640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00d0340 con 0x7fb0d00d2120 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0d5552640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d00d2c20 con 0x7fb0d006cb10 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 0x7fb0d0152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 0x7fb0d0152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38304/0 (socket says 172.21.15.156:38304) 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0cf7fe640 1 -- 172.21.15.156:0/2401948882 learned_addr learned my addr 172.21.15.156:0/2401948882 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0ce7fc640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d01f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.278+0000 7fb0ceffd640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d006cb10 0x7fb0d00b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0cf7fe640 1 -- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d006cb10 msgr2=0x7fb0d00b7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0cf7fe640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d006cb10 0x7fb0d00b7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0cf7fe640 1 -- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 msgr2=0x7fb0d01f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0cf7fe640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d01f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0cf7fe640 1 -- 172.21.15.156:0/2401948882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d01f5480 con 0x7fb0d015e920 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0cf7fe640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 0x7fb0d0152840 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800cbb0 tx=0x7fb0c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:56.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8017070 con 0x7fb0d015e920 2024-03-29T04:10:56.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c800ad40 con 0x7fb0d015e920 2024-03-29T04:10:56.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d01f5760 con 0x7fb0d015e920 2024-03-29T04:10:56.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8012600 con 0x7fb0d015e920 2024-03-29T04:10:56.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.279+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d01f5d20 con 0x7fb0d015e920 2024-03-29T04:10:56.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.280+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d00a2480 con 0x7fb0d015e920 2024-03-29T04:10:56.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.280+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb0c801a070 con 0x7fb0d015e920 2024-03-29T04:10:56.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.281+0000 7fb0b77fe640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0bc06c530 0x7fb0bc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.281+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fb0c808b3e0 con 0x7fb0d015e920 2024-03-29T04:10:56.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.281+0000 7fb0ceffd640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0bc06c530 0x7fb0bc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.281+0000 7fb0ceffd640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0bc06c530 0x7fb0bc06e9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb0d006d990 tx=0x7fb0c0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:56.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.285+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb0c80593b0 con 0x7fb0d015e920 2024-03-29T04:10:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.496+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb0d004f9d0 con 0x7fb0d015e920 2024-03-29T04:10:56.476 INFO:tasks.ceph.mon.a.smithi156.stderr:2024-03-29T04:10:56.497+0000 7fc37d67e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T04:10:56.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.498+0000 7fb0b77fe640 1 -- 172.21.15.156:0/2401948882 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7fb0c805d6b0 con 0x7fb0d015e920 2024-03-29T04:10:56.477 INFO:teuthology.orchestra.run.smithi156.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T04:10:56.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0bc06c530 msgr2=0x7fb0bc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0bc06c530 0x7fb0bc06e9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb0d006d990 tx=0x7fb0c0009190 comp rx=0 tx=0).stop 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 msgr2=0x7fb0d0152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 0x7fb0d0152840 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800cbb0 tx=0x7fb0c800a430 comp rx=0 tx=0).stop 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 shutdown_connections 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fb0bc06c530 0x7fb0bc06e9f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0d015e920 0x7fb0d0152840 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0d00d2120 0x7fb0d01f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 --2- 172.21.15.156:0/2401948882 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb0d006cb10 0x7fb0d00b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 >> 172.21.15.156:0/2401948882 conn(0x7fb0d009d7a0 msgr2=0x7fb0d00b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 shutdown_connections 2024-03-29T04:10:56.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.501+0000 7fb0d5552640 1 -- 172.21.15.156:0/2401948882 wait complete. 2024-03-29T04:10:56.496 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T04:10:56.522 INFO:tasks.ceph.mds.a.smithi156.stdout:starting mds.a at 2024-03-29T04:10:56.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.621+0000 7f0298504640 1 Processor -- start 2024-03-29T04:10:56.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.621+0000 7f0298504640 1 -- start start 2024-03-29T04:10:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0298504640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f02900d2020 0x7f02900d4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0298504640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900d26e0 con 0x7f02900cf4b0 2024-03-29T04:10:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0298504640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900d29c0 con 0x7f02900c78d0 2024-03-29T04:10:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0298504640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f02900d32f0 con 0x7f02900d2020 2024-03-29T04:10:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0295a9f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f02900d2020 0x7f02900d4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0296aa1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f02900cf4b0 0x7f02900d3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39822/0 (socket says 172.21.15.156:39822) 2024-03-29T04:10:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.622+0000 7f0296aa1640 1 -- 172.21.15.156:0/717575125 learned_addr learned my addr 172.21.15.156:0/717575125 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.623+0000 7f0295a9f640 1 -- 172.21.15.156:0/717575125 >> v1:172.21.15.203:6789/0 conn(0x7f02900c78d0 legacy=0x7f02900b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.623+0000 7f0295a9f640 1 -- 172.21.15.156:0/717575125 >> v1:172.21.15.156:6789/0 conn(0x7f02900cf4b0 legacy=0x7f02900d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.623+0000 7f0295a9f640 1 -- 172.21.15.156:0/717575125 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900d2ca0 con 0x7f02900d2020 2024-03-29T04:10:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.623+0000 7f029529e640 1 -- 172.21.15.156:0/717575125 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3455447678 0 0) 0x7f02900d26e0 con 0x7f02900cf4b0 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.623+0000 7f0295a9f640 1 --2- 172.21.15.156:0/717575125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900d2020 0x7f02900d4ef0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f02800097a0 tx=0x7f028002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517f1dcc9aff9cce server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f029529e640 1 -- 172.21.15.156:0/717575125 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280034070 con 0x7f02900d2020 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f029529e640 1 -- 172.21.15.156:0/717575125 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0280030610 con 0x7f02900d2020 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f029529e640 1 -- 172.21.15.156:0/717575125 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800309e0 con 0x7f02900d2020 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 -- 172.21.15.156:0/717575125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900d2020 msgr2=0x7f02900d4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 --2- 172.21.15.156:0/717575125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900d2020 0x7f02900d4ef0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f02800097a0 tx=0x7f028002b960 comp rx=0 tx=0).stop 2024-03-29T04:10:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 -- 172.21.15.156:0/717575125 shutdown_connections 2024-03-29T04:10:56.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 --2- 172.21.15.156:0/717575125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900d2020 0x7f02900d4ef0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 -- 172.21.15.156:0/717575125 >> 172.21.15.156:0/717575125 conn(0x7f029009d7a0 msgr2=0x7f02900d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:56.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 -- 172.21.15.156:0/717575125 shutdown_connections 2024-03-29T04:10:56.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.624+0000 7f0298504640 1 -- 172.21.15.156:0/717575125 wait complete. 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.625+0000 7f0298504640 1 Processor -- start 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.625+0000 7f0298504640 1 -- start start 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0298504640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 0x7f02900b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0298504640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02900cf4b0 0x7f029015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0298504640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02900d2020 0x7f02900b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0298504640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02900b1cd0 con 0x7f02900c78d0 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0298504640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f02900d2480 con 0x7f02900d2020 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0298504640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f02900c7cb0 con 0x7f02900cf4b0 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f02962a0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 0x7f02900b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f02962a0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 0x7f02900b9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38348/0 (socket says 172.21.15.156:38348) 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f02962a0640 1 -- 172.21.15.156:0/2176594082 learned_addr learned my addr 172.21.15.156:0/2176594082 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.626+0000 7f0296aa1640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02900d2020 0x7f02900b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f02962a0640 1 -- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02900cf4b0 msgr2=0x7f029015a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0295a9f640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02900cf4b0 0x7f029015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f02962a0640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02900cf4b0 0x7f029015a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f02962a0640 1 -- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02900d2020 msgr2=0x7f02900b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f02962a0640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02900d2020 0x7f02900b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f02962a0640 1 -- 172.21.15.156:0/2176594082 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900b9870 con 0x7f02900c78d0 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0296aa1640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02900d2020 0x7f02900b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f02962a0640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 0x7f02900b9330 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f028800c430 tx=0x7f028800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288017070 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02880052d0 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02900b8570 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288003830 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.627+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02900b8b30 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.628+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02900a2480 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.628+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f028801a070 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.629+0000 7f0286ffd640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f027406c530 0x7f027406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.629+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5795+0+0 (secure 0 0 0) 0x7f028808aa10 con 0x7f02900c78d0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.629+0000 7f0295a9f640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f027406c530 0x7f027406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:56.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.629+0000 7f0295a9f640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f027406c530 0x7f027406e9f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0280036000 tx=0x7f0280008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:56.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.633+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0288059060 con 0x7f02900c78d0 2024-03-29T04:10:56.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:56.846+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f02900cb8f0 con 0x7f02900c78d0 2024-03-29T04:10:57.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.499+0000 7f0286ffd640 1 -- 172.21.15.156:0/2176594082 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f0288058960 con 0x7f02900c78d0 2024-03-29T04:10:57.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f027406c530 msgr2=0x7f027406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:57.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f027406c530 0x7f027406e9f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0280036000 tx=0x7f0280008030 comp rx=0 tx=0).stop 2024-03-29T04:10:57.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 msgr2=0x7f02900b9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:57.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 0x7f02900b9330 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f028800c430 tx=0x7f028800c8f0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 shutdown_connections 2024-03-29T04:10:57.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f027406c530 0x7f027406e9f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02900d2020 0x7f02900b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02900cf4b0 0x7f029015a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 --2- 172.21.15.156:0/2176594082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02900c78d0 0x7f02900b9330 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 >> 172.21.15.156:0/2176594082 conn(0x7f029009d7a0 msgr2=0x7f02900d0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:57.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 shutdown_connections 2024-03-29T04:10:57.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.502+0000 7f0298504640 1 -- 172.21.15.156:0/2176594082 wait complete. 2024-03-29T04:10:57.493 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T04:10:57.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.660+0000 7f8a59772640 1 Processor -- start 2024-03-29T04:10:57.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.660+0000 7f8a59772640 1 -- start start 2024-03-29T04:10:57.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.660+0000 7f8a59772640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a5415c350 0x7f8a540d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:57.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.660+0000 7f8a59772640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a540d3110 0x7f8a540cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.660+0000 7f8a59772640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a5415ee40 con 0x7f8a540d1410 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a59772640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8a540d18c0 con 0x7f8a5415c350 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a59772640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5415f120 con 0x7f8a540d3110 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a52ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a5415c350 0x7f8a540d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a527fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8a540d1410 0x7f8a540d2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39824/0 (socket says 172.21.15.156:39824) 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a527fc640 1 -- 172.21.15.156:0/3772812347 learned_addr learned my addr 172.21.15.156:0/3772812347 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a537fe640 1 --2- 172.21.15.156:0/3772812347 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a540d3110 0x7f8a540cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3772812347 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a540d3110 msgr2=0x7f8a540cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a52ffd640 1 --2- 172.21.15.156:0/3772812347 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a540d3110 0x7f8a540cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3772812347 >> v1:172.21.15.156:6789/0 conn(0x7f8a540d1410 legacy=0x7f8a540d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3772812347 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5415f400 con 0x7f8a5415c350 2024-03-29T04:10:57.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.661+0000 7f8a51ffb640 1 -- 172.21.15.156:0/3772812347 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927128175 0 0) 0x7f8a5415ee40 con 0x7f8a540d1410 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a52ffd640 1 --2- 172.21.15.156:0/3772812347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a5415c350 0x7f8a540d0ed0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8a440097a0 tx=0x7f8a4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3e885735d9a35f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a51ffb640 1 -- 172.21.15.156:0/3772812347 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44035020 con 0x7f8a5415c350 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a51ffb640 1 -- 172.21.15.156:0/3772812347 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a44031830 con 0x7f8a5415c350 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a51ffb640 1 -- 172.21.15.156:0/3772812347 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44031c00 con 0x7f8a5415c350 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a59772640 1 -- 172.21.15.156:0/3772812347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a5415c350 msgr2=0x7f8a540d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a59772640 1 --2- 172.21.15.156:0/3772812347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a5415c350 0x7f8a540d0ed0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8a440097a0 tx=0x7f8a4402b8d0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a59772640 1 -- 172.21.15.156:0/3772812347 shutdown_connections 2024-03-29T04:10:57.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a59772640 1 --2- 172.21.15.156:0/3772812347 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a540d3110 0x7f8a540cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a59772640 1 --2- 172.21.15.156:0/3772812347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a5415c350 0x7f8a540d0ed0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.662+0000 7f8a59772640 1 -- 172.21.15.156:0/3772812347 >> 172.21.15.156:0/3772812347 conn(0x7f8a5408bfd0 msgr2=0x7f8a540d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:57.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.663+0000 7f8a59772640 1 -- 172.21.15.156:0/3772812347 shutdown_connections 2024-03-29T04:10:57.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.663+0000 7f8a59772640 1 -- 172.21.15.156:0/3772812347 wait complete. 2024-03-29T04:10:57.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 Processor -- start 2024-03-29T04:10:57.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 -- start start 2024-03-29T04:10:57.646 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 0x7f8a540b9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a540d3110 0x7f8a540b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a5415c350 0x7f8a54159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a540cf980 con 0x7f8a540d1410 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a540cf800 con 0x7f8a540d3110 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.664+0000 7f8a59772640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f8a540cfb00 con 0x7f8a5415c350 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 0x7f8a540b9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 0x7f8a540b9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38368/0 (socket says 172.21.15.156:38368) 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3838435147 learned_addr learned my addr 172.21.15.156:0/3838435147 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a537fe640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a5415c350 0x7f8a54159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a527fc640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a540d3110 0x7f8a540b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a5415c350 msgr2=0x7f8a54159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a5415c350 0x7f8a54159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a540d3110 msgr2=0x7f8a540b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a540d3110 0x7f8a540b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 -- 172.21.15.156:0/3838435147 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a540b7610 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a527fc640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a540d3110 0x7f8a540b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a52ffd640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 0x7f8a540b9070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8a440098c0 tx=0x7f8a440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44040070 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4403e070 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a540b78b0 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44035040 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.665+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a540b7e70 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.666+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a540730b0 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.667+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8a44038020 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.667+0000 7f8a3b7fe640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8a4006c560 0x7f8a4006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.667+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5795+0+0 (secure 0 0 0) 0x7f8a440acd20 con 0x7f8a540d1410 2024-03-29T04:10:57.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.667+0000 7f8a527fc640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8a4006c560 0x7f8a4006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:57.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.668+0000 7f8a527fc640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8a4006c560 0x7f8a4006ea20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8a541619b0 tx=0x7f8a48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:57.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.672+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8a540730b0 con 0x7f8a540d1410 2024-03-29T04:10:57.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:57.845+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8a54068d20 con 0x7f8a540d1410 2024-03-29T04:10:58.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.504+0000 7f8a3b7fe640 1 -- 172.21.15.156:0/3838435147 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) ==== 181+0+0 (secure 0 0 0) 0x7f8a4407b2f0 con 0x7f8a540d1410 2024-03-29T04:10:58.483 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_metadata' removed 2024-03-29T04:10:58.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8a4006c560 msgr2=0x7f8a4006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:58.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8a4006c560 0x7f8a4006ea20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8a541619b0 tx=0x7f8a48009040 comp rx=0 tx=0).stop 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 msgr2=0x7f8a540b9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 0x7f8a540b9070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8a440098c0 tx=0x7f8a440046c0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 shutdown_connections 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f8a4006c560 0x7f8a4006ea20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a5415c350 0x7f8a54159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a540d3110 0x7f8a540b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 --2- 172.21.15.156:0/3838435147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a540d1410 0x7f8a540b9070 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 >> 172.21.15.156:0/3838435147 conn(0x7f8a5408bfd0 msgr2=0x7f8a540cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 shutdown_connections 2024-03-29T04:10:58.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.507+0000 7f8a59772640 1 -- 172.21.15.156:0/3838435147 wait complete. 2024-03-29T04:10:58.499 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T04:10:58.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.628+0000 7f447a367640 1 Processor -- start 2024-03-29T04:10:58.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.628+0000 7f447a367640 1 -- start start 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f447a367640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f447415b460 0x7f44740ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f447a367640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44740729b0 con 0x7f4474071810 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f447a367640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4474072c90 con 0x7f44740d0b80 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f447a367640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f4474072f70 con 0x7f447415b460 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f4473fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f447415b460 0x7f44740ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f44737fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4474071810 0x7f44740d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39830/0 (socket says 172.21.15.156:39830) 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f4473fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f447415b460 0x7f44740ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38378/0 (socket says 172.21.15.156:38378) 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.629+0000 7f44737fe640 1 -- 172.21.15.156:0/3217872878 learned_addr learned my addr 172.21.15.156:0/3217872878 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4473fff640 1 -- 172.21.15.156:0/3217872878 >> v1:172.21.15.203:6789/0 conn(0x7f44740d0b80 legacy=0x7f44740b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4473fff640 1 -- 172.21.15.156:0/3217872878 >> v1:172.21.15.156:6789/0 conn(0x7f4474071810 legacy=0x7f44740d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4473fff640 1 -- 172.21.15.156:0/3217872878 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4474072680 con 0x7f447415b460 2024-03-29T04:10:58.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4472ffd640 1 -- 172.21.15.156:0/3217872878 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778040128 0 0) 0x7f44740729b0 con 0x7f4474071810 2024-03-29T04:10:58.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4473fff640 1 --2- 172.21.15.156:0/3217872878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447415b460 0x7f44740ada60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f44640097a0 tx=0x7f446402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98bba4f82be4b2d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4472ffd640 1 -- 172.21.15.156:0/3217872878 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464038070 con 0x7f447415b460 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4472ffd640 1 -- 172.21.15.156:0/3217872878 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4464031700 con 0x7f447415b460 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.630+0000 7f4472ffd640 1 -- 172.21.15.156:0/3217872878 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464031ad0 con 0x7f447415b460 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 -- 172.21.15.156:0/3217872878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447415b460 msgr2=0x7f44740ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 --2- 172.21.15.156:0/3217872878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447415b460 0x7f44740ada60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f44640097a0 tx=0x7f446402b7a0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 -- 172.21.15.156:0/3217872878 shutdown_connections 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 --2- 172.21.15.156:0/3217872878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447415b460 0x7f44740ada60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 -- 172.21.15.156:0/3217872878 >> 172.21.15.156:0/3217872878 conn(0x7f44740b21e0 msgr2=0x7f44740b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 -- 172.21.15.156:0/3217872878 shutdown_connections 2024-03-29T04:10:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.631+0000 7f447a367640 1 -- 172.21.15.156:0/3217872878 wait complete. 2024-03-29T04:10:58.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.632+0000 7f447a367640 1 Processor -- start 2024-03-29T04:10:58.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.632+0000 7f447a367640 1 -- start start 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f447a367640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4474071810 0x7f4474148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f447a367640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 0x7f4474148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f447a367640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f447415b460 0x7f44741f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f447a367640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4474071e10 con 0x7f44740d0b80 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f447a367640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4474071c90 con 0x7f4474071810 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f447a367640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f4474071f90 con 0x7f447415b460 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f44737fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 0x7f4474148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f44737fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 0x7f4474148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38386/0 (socket says 172.21.15.156:38386) 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f44737fe640 1 -- 172.21.15.156:0/3214853648 learned_addr learned my addr 172.21.15.156:0/3214853648 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f4473fff640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4474071810 0x7f4474148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f44737fe640 1 -- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f447415b460 msgr2=0x7f44741f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.633+0000 7f44737fe640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f447415b460 0x7f44741f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f44737fe640 1 -- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4474071810 msgr2=0x7f4474148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f44737fe640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4474071810 0x7f4474148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f44737fe640 1 -- 172.21.15.156:0/3214853648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447409dc50 con 0x7f44740d0b80 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f44737fe640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 0x7f4474148db0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f446800c3c0 tx=0x7f446800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468017070 con 0x7f44740d0b80 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4468005ce0 con 0x7f44740d0b80 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447409df30 con 0x7f44740d0b80 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468007310 con 0x7f44740d0b80 2024-03-29T04:10:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.634+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44740991b0 con 0x7f44740d0b80 2024-03-29T04:10:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.635+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4474068610 con 0x7f44740d0b80 2024-03-29T04:10:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.635+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f446801a070 con 0x7f44740d0b80 2024-03-29T04:10:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.636+0000 7f4470ff9640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f446006c560 0x7f446006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.636+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5357+0+0 (secure 0 0 0) 0x7f446808a910 con 0x7f44740d0b80 2024-03-29T04:10:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.636+0000 7f4473fff640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f446006c560 0x7f446006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.636+0000 7f4473fff640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f446006c560 0x7f446006ea20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4464035000 tx=0x7f4464008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:58.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.640+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4468059090 con 0x7f44740d0b80 2024-03-29T04:10:58.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:58.816+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f447406f3f0 con 0x7f44740d0b80 2024-03-29T04:10:59.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.505+0000 7f4470ff9640 1 -- 172.21.15.156:0/3214853648 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) ==== 169+0+0 (secure 0 0 0) 0x7f4468058990 con 0x7f44740d0b80 2024-03-29T04:10:59.484 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_data' removed 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f446006c560 msgr2=0x7f446006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f446006c560 0x7f446006ea20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4464035000 tx=0x7f4464008030 comp rx=0 tx=0).stop 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 msgr2=0x7f4474148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 0x7f4474148db0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f446800c3c0 tx=0x7f446800c880 comp rx=0 tx=0).stop 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 shutdown_connections 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f446006c560 0x7f446006ea20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f447415b460 0x7f44741f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44740d0b80 0x7f4474148db0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 --2- 172.21.15.156:0/3214853648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4474071810 0x7f4474148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.509+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 >> 172.21.15.156:0/3214853648 conn(0x7f44740b21e0 msgr2=0x7f447406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:59.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.510+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 shutdown_connections 2024-03-29T04:10:59.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.510+0000 7f447a367640 1 -- 172.21.15.156:0/3214853648 wait complete. 2024-03-29T04:10:59.504 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T04:10:59.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.632+0000 7f7b909af640 1 Processor -- start 2024-03-29T04:10:59.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.632+0000 7f7b909af640 1 -- start start 2024-03-29T04:10:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b909af640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7b8c15e540 0x7f7b8c0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b909af640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b8c0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b909af640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8c0d0020 con 0x7f7b8c0cfb90 2024-03-29T04:10:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b909af640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7b8c06c970 con 0x7f7b8c15e540 2024-03-29T04:10:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b909af640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8c0d2fd0 con 0x7f7b8c0d2a00 2024-03-29T04:10:59.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b8ad9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7b8c0cfb90 0x7f7b8c0d0d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39844/0 (socket says 172.21.15.156:39844) 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.633+0000 7f7b8ad9d640 1 -- 172.21.15.156:0/187391166 learned_addr learned my addr 172.21.15.156:0/187391166 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/187391166 >> v2:172.21.15.203:3300/0 conn(0x7f7b8c15e540 0x7f7b8c0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b89d9b640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b8c0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364162496 0 0) 0x7f7b8c0d0020 con 0x7f7b8c0cfb90 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b74003520 con 0x7f7b8c0cfb90 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1190296049 0 0) 0x7f7b74003520 con 0x7f7b8c0cfb90 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 msgr2=0x7f7b8c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b8c0cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 >> v2:172.21.15.203:3300/0 conn(0x7f7b8c15e540 msgr2=0x7f7b8c0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 --2- 172.21.15.156:0/187391166 >> v2:172.21.15.203:3300/0 conn(0x7f7b8c15e540 0x7f7b8c0cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c0d32b0 con 0x7f7b8c0cfb90 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7f7b84001670 con 0x7f7b8c0cfb90 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/187391166 >> v2:172.21.15.203:3300/0 conn(0x7f7b8c15e540 0x7f7b8c0cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 >> v1:172.21.15.156:6789/0 conn(0x7f7b8c0cfb90 legacy=0x7f7b8c0d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b740039b0 0x7f7b74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b74004800 0x7f7b74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c06c970 con 0x7f7b740039b0 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c0d2fd0 con 0x7f7b74004800 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7400b9c0 con 0x7f7b8c0d2a00 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b740039b0 0x7f7b74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b89d9b640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b74004800 0x7f7b74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8ad9d640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 -- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 msgr2=0x7f7b740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 -- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b74004800 msgr2=0x7f7b74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b74004800 0x7f7b74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8a59c640 1 -- 172.21.15.156:0/187391166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b74003520 con 0x7f7b740039b0 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b8ad9d640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0d2a00 0x7f7b740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:10:59.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.634+0000 7f7b89d9b640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b74004800 0x7f7b74004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.635+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b740039b0 0x7f7b74003dd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e350 tx=0x7f7b74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d139b59d02d9a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.635+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c039070 con 0x7f7b740039b0 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.635+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7c02ce00 con 0x7f7b740039b0 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.635+0000 7f7b8959a640 1 -- 172.21.15.156:0/187391166 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c009de0 con 0x7f7b740039b0 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b909af640 1 -- 172.21.15.156:0/187391166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b740039b0 msgr2=0x7f7b74003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b909af640 1 --2- 172.21.15.156:0/187391166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b740039b0 0x7f7b74003dd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e350 tx=0x7f7b74005830 comp rx=0 tx=0).stop 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b8ad9d640 1 -- 172.21.15.156:0/187391166 reap_dead start 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b909af640 1 -- 172.21.15.156:0/187391166 shutdown_connections 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b909af640 1 -- 172.21.15.156:0/187391166 >> 172.21.15.156:0/187391166 conn(0x7f7b8c0d4450 msgr2=0x7f7b8c0d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b909af640 1 -- 172.21.15.156:0/187391166 shutdown_connections 2024-03-29T04:10:59.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.636+0000 7f7b909af640 1 -- 172.21.15.156:0/187391166 wait complete. 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.637+0000 7f7b909af640 1 Processor -- start 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.637+0000 7f7b909af640 1 -- start start 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b909af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b740039b0 0x7f7b8c0c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b909af640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 0x7f7b8c14e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b909af640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0b9510 0x7f7b8c0b9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b909af640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c15eaa0 con 0x7f7b8c15ee50 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b909af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c15e920 con 0x7f7b740039b0 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b909af640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8c06de10 con 0x7f7b8c0b9510 2024-03-29T04:10:59.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 0x7f7b8c14e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 0x7f7b8c14e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38404/0 (socket says 172.21.15.156:38404) 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 -- 172.21.15.156:0/2775618277 learned_addr learned my addr 172.21.15.156:0/2775618277 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b8ad9d640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0b9510 0x7f7b8c0b9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b740039b0 0x7f7b8c0c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 -- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0b9510 msgr2=0x7f7b8c0b9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0b9510 0x7f7b8c0b9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 -- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b740039b0 msgr2=0x7f7b8c0c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b740039b0 0x7f7b8c0c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b89d9b640 1 -- 172.21.15.156:0/2775618277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c0c95f0 con 0x7f7b8c15ee50 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b8ad9d640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0b9510 0x7f7b8c0b9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.638+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b740039b0 0x7f7b8c0c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.639+0000 7f7b89d9b640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 0x7f7b8c14e0a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7b80011700 tx=0x7f7b80011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.639+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000f9c0 con 0x7f7b8c15ee50 2024-03-29T04:10:59.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.639+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8000fb60 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.639+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c0c98d0 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.639+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b80015460 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.639+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c0c9e90 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.640+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8c0a23a0 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.640+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7b80015600 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.641+0000 7f7b7affd640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b6c06c560 0x7f7b6c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.641+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f7b8008bc70 con 0x7f7b8c15ee50 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.641+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b6c06c560 0x7f7b6c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.641+0000 7f7b8a59c640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b6c06c560 0x7f7b6c06ea20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c036000 tx=0x7f7b7c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:59.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.645+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7b80019030 con 0x7f7b8c15ee50 2024-03-29T04:10:59.803 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.824+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f7b8c04f9d0 con 0x7f7b8c15ee50 2024-03-29T04:10:59.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.827+0000 7f7b7affd640 1 -- 172.21.15.156:0/2775618277 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f7b8005a620 con 0x7f7b8c15ee50 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b6c06c560 msgr2=0x7f7b6c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b6c06c560 0x7f7b6c06ea20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c036000 tx=0x7f7b7c02e000 comp rx=0 tx=0).stop 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 msgr2=0x7f7b8c14e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 0x7f7b8c14e0a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7b80011700 tx=0x7f7b80011bc0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 shutdown_connections 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b6c06c560 0x7f7b6c06ea20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b8c0b9510 0x7f7b8c0b9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c15ee50 0x7f7b8c14e0a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 --2- 172.21.15.156:0/2775618277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b740039b0 0x7f7b8c0c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.829+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 >> 172.21.15.156:0/2775618277 conn(0x7f7b8c0d4450 msgr2=0x7f7b8c0d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.830+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 shutdown_connections 2024-03-29T04:10:59.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.830+0000 7f7b909af640 1 -- 172.21.15.156:0/2775618277 wait complete. 2024-03-29T04:10:59.820 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_uninlining.TestDataUninlining.test_data_uninlining' 2024-03-29T04:10:59.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.941+0000 7ff28563c640 1 Processor -- start 2024-03-29T04:10:59.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.941+0000 7ff28563c640 1 -- start start 2024-03-29T04:10:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff28563c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff28015e750 0x7ff2800b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff28563c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff28006b5b0 0x7ff28006ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff28563c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006e870 0x7ff28006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff28563c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff28006fa90 con 0x7ff28006b5b0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff28563c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff28006fce0 con 0x7ff28015e750 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff28563c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff28006ffc0 con 0x7ff28006e870 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff28006b5b0 0x7ff28006ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff28006b5b0 0x7ff28006ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38406/0 (socket says 172.21.15.156:38406) 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 -- 172.21.15.156:0/223427166 learned_addr learned my addr 172.21.15.156:0/223427166 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27f7fe640 1 --2- 172.21.15.156:0/223427166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006e870 0x7ff28006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27effd640 1 --2- 172.21.15.156:0/223427166 >> v2:172.21.15.203:3300/0 conn(0x7ff28015e750 0x7ff2800b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 -- 172.21.15.156:0/223427166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006e870 msgr2=0x7ff28006ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 --2- 172.21.15.156:0/223427166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006e870 0x7ff28006ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 -- 172.21.15.156:0/223427166 >> v2:172.21.15.203:3300/0 conn(0x7ff28015e750 msgr2=0x7ff2800b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 --2- 172.21.15.156:0/223427166 >> v2:172.21.15.203:3300/0 conn(0x7ff28015e750 0x7ff2800b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.942+0000 7ff27e7fc640 1 -- 172.21.15.156:0/223427166 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2800702a0 con 0x7ff28006b5b0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.943+0000 7ff27e7fc640 1 --2- 172.21.15.156:0/223427166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28006b5b0 0x7ff28006ba40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff280073c50 tx=0x7ff27402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4e92310f871add server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.943+0000 7ff27dffb640 1 -- 172.21.15.156:0/223427166 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff274035020 con 0x7ff28006b5b0 2024-03-29T04:10:59.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.943+0000 7ff27dffb640 1 -- 172.21.15.156:0/223427166 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff274031870 con 0x7ff28006b5b0 2024-03-29T04:10:59.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.943+0000 7ff27dffb640 1 -- 172.21.15.156:0/223427166 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff274031c40 con 0x7ff28006b5b0 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 -- 172.21.15.156:0/223427166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28006b5b0 msgr2=0x7ff28006ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 --2- 172.21.15.156:0/223427166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28006b5b0 0x7ff28006ba40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff280073c50 tx=0x7ff27402b910 comp rx=0 tx=0).stop 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 -- 172.21.15.156:0/223427166 shutdown_connections 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 --2- 172.21.15.156:0/223427166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28006b5b0 0x7ff28006ba40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 --2- 172.21.15.156:0/223427166 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006e870 0x7ff28006ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 --2- 172.21.15.156:0/223427166 >> v2:172.21.15.203:3300/0 conn(0x7ff28015e750 0x7ff2800b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 -- 172.21.15.156:0/223427166 >> 172.21.15.156:0/223427166 conn(0x7ff280072d10 msgr2=0x7ff2800b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:10:59.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 -- 172.21.15.156:0/223427166 shutdown_connections 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.944+0000 7ff28563c640 1 -- 172.21.15.156:0/223427166 wait complete. 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.945+0000 7ff28563c640 1 Processor -- start 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.945+0000 7ff28563c640 1 -- start start 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff28563c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006b5b0 0x7ff28009bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff28563c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff28006e870 0x7ff28009e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff28563c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 0x7ff280148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff28563c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff28009ac60 con 0x7ff28015e750 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff28563c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff280070860 con 0x7ff28006e870 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff28563c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7ff28015eb30 con 0x7ff28006b5b0 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 0x7ff280148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 0x7ff280148bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:38412/0 (socket says 172.21.15.156:38412) 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 -- 172.21.15.156:0/1996481165 learned_addr learned my addr 172.21.15.156:0/1996481165 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 -- 172.21.15.156:0/1996481165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006b5b0 msgr2=0x7ff28009bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006b5b0 0x7ff28009bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 -- 172.21.15.156:0/1996481165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff28006e870 msgr2=0x7ff28009e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff28006e870 0x7ff28009e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.946+0000 7ff27f7fe640 1 -- 172.21.15.156:0/1996481165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff280149860 con 0x7ff28015e750 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.947+0000 7ff27f7fe640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 0x7ff280148bf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff278007e60 tx=0x7ff27800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.947+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278014620 con 0x7ff28015e750 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.947+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff278019070 con 0x7ff28015e750 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.947+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff280149b20 con 0x7ff28015e750 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.947+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278014a20 con 0x7ff28015e750 2024-03-29T04:10:59.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.947+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28014a0e0 con 0x7ff28015e750 2024-03-29T04:10:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.948+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2800b49e0 con 0x7ff28015e750 2024-03-29T04:10:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.948+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff278014bc0 con 0x7ff28015e750 2024-03-29T04:10:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.948+0000 7ff2677fe640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff26c06c560 0x7ff26c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:10:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.949+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7ff27808bc20 con 0x7ff28015e750 2024-03-29T04:10:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.949+0000 7ff27effd640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff26c06c560 0x7ff26c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:10:59.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.949+0000 7ff27effd640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff26c06c560 0x7ff26c06ea20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff270006fd0 tx=0x7ff270008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:10:59.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:10:59.953+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff2800b49e0 con 0x7ff28015e750 2024-03-29T04:11:00.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.198+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_uninlining.TestDataUninlining.test_data_uninlining"]} v 0) -- 0x7ff28006bb70 con 0x7ff28015e750 2024-03-29T04:11:00.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.509+0000 7ff2677fe640 1 -- 172.21.15.156:0/1996481165 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_uninlining.TestDataUninlining.test_data_uninlining"]}]=0 v130) ==== 147+0+0 (secure 0 0 0) 0x7ff27805a540 con 0x7ff28015e750 2024-03-29T04:11:00.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff26c06c560 msgr2=0x7ff26c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:00.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff26c06c560 0x7ff26c06ea20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff270006fd0 tx=0x7ff270008040 comp rx=0 tx=0).stop 2024-03-29T04:11:00.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 msgr2=0x7ff280148bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:00.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 0x7ff280148bf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff278007e60 tx=0x7ff27800e4b0 comp rx=0 tx=0).stop 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 shutdown_connections 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7ff26c06c560 0x7ff26c06ea20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff28015e750 0x7ff280148bf0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff28006e870 0x7ff28009e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 --2- 172.21.15.156:0/1996481165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff28006b5b0 0x7ff28009bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 >> 172.21.15.156:0/1996481165 conn(0x7ff280072d10 msgr2=0x7ff28015ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 shutdown_connections 2024-03-29T04:11:00.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:00.511+0000 7ff28563c640 1 -- 172.21.15.156:0/1996481165 wait complete. 2024-03-29T04:11:00.504 INFO:tasks.cephfs_test_runner:test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) ... ERROR 2024-03-29T04:11:00.504 INFO:tasks.cephfs_test_runner: 2024-03-29T04:11:00.504 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-29T04:11:00.504 INFO:tasks.cephfs_test_runner:ERROR: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2024-03-29T04:11:00.504 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 322, in test_data_uninlining 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: self.run_test_worker(opt_clear_cache, 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 259, in run_test_worker 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: info = self.get_inline_data_info(inodes, files, dir_pins, num_mds) 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 116, in get_inline_data_info 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: json_out = get_inode_dump(inode, 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 101, in get_inode_dump 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: json_out = self.fs.rank_tell(['dump', 'inode', inode], rank) 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/filesystem.py", line 1332, in rank_tell 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/filesystem.py", line 1319, in mds_tell 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-03-29T04:11:00.505 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 447, in run 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: r = RemoteProcess(client, args, check_status=check_status, hostname=name, 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 65, in __init__ 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: self.command = quote(args) 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 257, in quote 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: return ' '.join(_quote(args)) 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 255, in _quote 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: yield pipes.quote(a) 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: File "/usr/lib/python3.8/shlex.py", line 325, in quote 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: if _find_unsafe(s) is None: 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner:TypeError: expected string or bytes-like object 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner: 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-29T04:11:00.506 INFO:tasks.cephfs_test_runner:Ran 1 test in 97.863s 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner:ERROR: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 322, in test_data_uninlining 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: self.run_test_worker(opt_clear_cache, 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 259, in run_test_worker 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: info = self.get_inline_data_info(inodes, files, dir_pins, num_mds) 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 116, in get_inline_data_info 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: json_out = get_inode_dump(inode, 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/test_uninlining.py", line 101, in get_inode_dump 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: json_out = self.fs.rank_tell(['dump', 'inode', inode], rank) 2024-03-29T04:11:00.507 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/filesystem.py", line 1332, in rank_tell 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs/filesystem.py", line 1319, in mds_tell 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 447, in run 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: r = RemoteProcess(client, args, check_status=check_status, hostname=name, 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 65, in __init__ 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: self.command = quote(args) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 257, in quote 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: return ' '.join(_quote(args)) 2024-03-29T04:11:00.508 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 255, in _quote 2024-03-29T04:11:00.509 INFO:tasks.cephfs_test_runner: yield pipes.quote(a) 2024-03-29T04:11:00.509 INFO:tasks.cephfs_test_runner: File "/usr/lib/python3.8/shlex.py", line 325, in quote 2024-03-29T04:11:00.509 INFO:tasks.cephfs_test_runner: if _find_unsafe(s) is None: 2024-03-29T04:11:00.509 INFO:tasks.cephfs_test_runner:TypeError: expected string or bytes-like object 2024-03-29T04:11:00.509 INFO:tasks.cephfs_test_runner: 2024-03-29T04:11:00.509 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2024-03-29T04:11:00.752 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c5732608d4b84a2da0647cab8b5064b8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_69e2cd73c52284dfd5096c66b60c6e6b26a3c447/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2024-03-29T04:11:00.754 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-29T04:11:00.763 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-29T04:11:00.804 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-29T04:11:00.805 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:11:00.805 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:11:00.825 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17587 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-29T04:11:00.825 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-29T04:11:00.826 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:11:00.881 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17587 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-29T04:11:00.881 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:11:00.881 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:11:00.900 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056352,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=427 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-29T04:11:00.901 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T04:11:00.901 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T04:11:00.959 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056352,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=427 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-29T04:11:00.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:00.959 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-03-29T04:11:01.017 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-29T04:11:01.017 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-29T04:11:01.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T04:11:01.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T04:11:01.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link delete ceph-brx 2024-03-29T04:11:01.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.258 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-03-29T04:11:01.273 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T04:11:01.273 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T04:11:01.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.274 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T04:11:01.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.342 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T04:11:01.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T04:11:01.411 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2024-03-29T04:11:01.510 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T04:11:01.519 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-29T04:11:01.520 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:11:01.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.642+0000 7fd17aa37640 1 Processor -- start 2024-03-29T04:11:01.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.642+0000 7fd17aa37640 1 -- start start 2024-03-29T04:11:01.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd17aa37640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 0x7fd174142690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd17aa37640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1740b30f0 con 0x7fd1740ad740 2024-03-29T04:11:01.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd17aa37640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17415d020 con 0x7fd17415c650 2024-03-29T04:11:01.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd17aa37640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd174142bd0 con 0x7fd17415c270 2024-03-29T04:11:01.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd178fd4640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd1740ad740 0x7fd1740b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54664/0 (socket says 172.21.15.156:54664) 2024-03-29T04:11:01.629 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd178fd4640 1 -- 172.21.15.156:0/2332878937 learned_addr learned my addr 172.21.15.156:0/2332878937 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:01.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22972711 0 0) 0x7fd1740b30f0 con 0x7fd1740ad740 2024-03-29T04:11:01.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd173fff640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 0x7fd174142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd160003520 con 0x7fd1740ad740 2024-03-29T04:11:01.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 743631694 0 0) 0x7fd160003520 con 0x7fd1740ad740 2024-03-29T04:11:01.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 msgr2=0x7fd174142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:01.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 0x7fd174142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 >> v1:172.21.15.203:6789/0 conn(0x7fd17415c650 legacy=0x7fd1740ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17415d300 con 0x7fd1740ad740 2024-03-29T04:11:01.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fd16c0019f0 con 0x7fd1740ad740 2024-03-29T04:11:01.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 >> v1:172.21.15.156:6789/0 conn(0x7fd1740ad740 legacy=0x7fd1740b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.643+0000 7fd1737fe640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 0x7fd1600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd1737fe640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd160004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd1737fe640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd160004d40 0x7fd1600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd174142bd0 con 0x7fd160003f20 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd16000bad0 con 0x7fd160004d40 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd16000bcd0 con 0x7fd17415c270 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd16bfff640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd160004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd173fff640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 0x7fd1600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd178fd4640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd160004d40 0x7fd1600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd16bfff640 1 -- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 msgr2=0x7fd1600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd16bfff640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd17415c270 0x7fd1600039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd16bfff640 1 -- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd160004d40 msgr2=0x7fd1600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd16bfff640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd160004d40 0x7fd1600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd16bfff640 1 -- 172.21.15.156:0/2332878937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd160003520 con 0x7fd160003f20 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.644+0000 7fd178fd4640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd160004d40 0x7fd1600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd16bfff640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd160004800 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd15c001860 tx=0x7fd160005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5062a83e389db4f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:01.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c03d070 con 0x7fd160003f20 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15c030790 con 0x7fd160003f20 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd1737fe640 1 -- 172.21.15.156:0/2332878937 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c001430 con 0x7fd160003f20 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd17aa37640 1 -- 172.21.15.156:0/2332878937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 msgr2=0x7fd160004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd17aa37640 1 --2- 172.21.15.156:0/2332878937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd160004800 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd15c001860 tx=0x7fd160005830 comp rx=0 tx=0).stop 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd178fd4640 1 -- 172.21.15.156:0/2332878937 reap_dead start 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd17aa37640 1 -- 172.21.15.156:0/2332878937 shutdown_connections 2024-03-29T04:11:01.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd17aa37640 1 -- 172.21.15.156:0/2332878937 >> 172.21.15.156:0/2332878937 conn(0x7fd1740b1d50 msgr2=0x7fd1740b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd17aa37640 1 -- 172.21.15.156:0/2332878937 shutdown_connections 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.645+0000 7fd17aa37640 1 -- 172.21.15.156:0/2332878937 wait complete. 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.647+0000 7fd17aa37640 1 Processor -- start 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.647+0000 7fd17aa37640 1 -- start start 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd174099b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd17415e590 0x7fd1741436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd174154810 0x7fd174154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd17415cad0 con 0x7fd160003f20 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd17415d8c0 con 0x7fd17415e590 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd17415cc50 con 0x7fd174154810 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd174099b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd174099b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35890/0 (socket says 172.21.15.156:35890) 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 -- 172.21.15.156:0/1742619272 learned_addr learned my addr 172.21.15.156:0/1742619272 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd16bfff640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd17415e590 0x7fd1741436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd178fd4640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd174154810 0x7fd174154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 -- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd174154810 msgr2=0x7fd174154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd174154810 0x7fd174154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 -- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd17415e590 msgr2=0x7fd1741436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd17415e590 0x7fd1741436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 -- 172.21.15.156:0/1742619272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1740d07c0 con 0x7fd160003f20 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd173fff640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd174099b20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd1640119a0 tx=0x7fd164011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16400faf0 con 0x7fd160003f20 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16400fc90 con 0x7fd160003f20 2024-03-29T04:11:01.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1740d0aa0 con 0x7fd160003f20 2024-03-29T04:11:01.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.648+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164014440 con 0x7fd160003f20 2024-03-29T04:11:01.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.649+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd174143be0 con 0x7fd160003f20 2024-03-29T04:11:01.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.650+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd174068610 con 0x7fd160003f20 2024-03-29T04:11:01.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.650+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd164012040 con 0x7fd160003f20 2024-03-29T04:11:01.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.650+0000 7fd1717fa640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd15406c530 0x7fd15406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.650+0000 7fd16bfff640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd15406c530 0x7fd15406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.650+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fd16408b960 con 0x7fd160003f20 2024-03-29T04:11:01.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.651+0000 7fd16bfff640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd15406c530 0x7fd15406e9f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd15c03a000 tx=0x7fd15c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:01.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.655+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd164019030 con 0x7fd160003f20 2024-03-29T04:11:01.807 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.828+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1740b4d60 con 0x7fd15406c530 2024-03-29T04:11:01.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.829+0000 7fd1717fa640 1 -- 172.21.15.156:0/1742619272 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13526 (secure 0 0 0) 0x7fd1740b4d60 con 0x7fd15406c530 2024-03-29T04:11:01.808 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:01.809 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:11:01.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd15406c530 msgr2=0x7fd15406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd15406c530 0x7fd15406e9f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd15c03a000 tx=0x7fd15c009030 comp rx=0 tx=0).stop 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 msgr2=0x7fd174099b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd174099b20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd1640119a0 tx=0x7fd164011e60 comp rx=0 tx=0).stop 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 shutdown_connections 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd15406c530 0x7fd15406e9f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd174154810 0x7fd174154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd17415e590 0x7fd1741436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 --2- 172.21.15.156:0/1742619272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd160003f20 0x7fd174099b20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 >> 172.21.15.156:0/1742619272 conn(0x7fd1740b1d50 msgr2=0x7fd17415dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:01.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 shutdown_connections 2024-03-29T04:11:01.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.831+0000 7fd17aa37640 1 -- 172.21.15.156:0/1742619272 wait complete. 2024-03-29T04:11:01.822 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-03-29T04:11:01.659933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":552956,"kb_used_data":320168,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754421764,"statfs":{"total":773094113280,"available":772527886336,"internally_reserved":0,"allocated":327852032,"data_stored":325556867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-174309431,"num_objects":-138,"num_object_clones":0,"num_object_copies":-276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-138,"num_whiteouts":0,"num_read":-43,"num_read_kb":-2,"num_write":-332,"num_write_kb":-170316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000913"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022723+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:09:55.022723+0000","last_peered":"2024-03-29T04:09:55.022723+0000","last_clean":"2024-03-29T04:09:55.022723+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:09:55.022723+0000","last_undegraded":"2024-03-29T04:09:55.022723+0000","last_fullsized":"2024-03-29T04:09:55.022723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542174,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73436,"kb_used_data":46420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94298404,"statfs":{"total":96636764160,"available":96561565696,"internally_reserved":0,"allocated":47534080,"data_stored":47246895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52096,"kb_used_data":25080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94319744,"statfs":{"total":96636764160,"available":96583417856,"internally_reserved":0,"allocated":25681920,"data_stored":25422162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542175,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89104,"kb_used_data":62088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94282736,"statfs":{"total":96636764160,"available":96545521664,"internally_reserved":0,"allocated":63578112,"data_stored":63292157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542175,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85748,"kb_used_data":58732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94286092,"statfs":{"total":96636764160,"available":96548958208,"internally_reserved":0,"allocated":60141568,"data_stored":59858595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69504,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":39243776,"data_stored":38912610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31600,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86240,"kb_used_data":55060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94285600,"statfs":{"total":96636764160,"available":96548454400,"internally_reserved":0,"allocated":56381440,"data_stored":56075612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65228,"kb_used_data":34048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306612,"statfs":{"total":96636764160,"available":96569970688,"internally_reserved":0,"allocated":34865152,"data_stored":34581028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:11:01.822 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:11:01.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.944+0000 7f62acc14640 1 Processor -- start 2024-03-29T04:11:01.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.944+0000 7f62acc14640 1 -- start start 2024-03-29T04:11:01.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62acc14640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a80af6b0 0x7f62a815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62acc14640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f62a806f7a0 0x7f62a8071750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62acc14640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62a815c7e0 con 0x7f62a80b3050 2024-03-29T04:11:01.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62acc14640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f62a80afa90 con 0x7f62a806f7a0 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62acc14640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f62a80b3510 con 0x7f62a80af6b0 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f62a806f7a0 0x7f62a8071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f62a806f7a0 0x7f62a8071750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35900/0 (socket says 172.21.15.156:35900) 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 -- 172.21.15.156:0/1564829198 learned_addr learned my addr 172.21.15.156:0/1564829198 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f629ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f62a80b3050 0x7f62a806f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54668/0 (socket says 172.21.15.156:54668) 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 -- 172.21.15.156:0/1564829198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a80af6b0 msgr2=0x7f62a815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 --2- 172.21.15.156:0/1564829198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a80af6b0 0x7f62a815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 -- 172.21.15.156:0/1564829198 >> v1:172.21.15.156:6789/0 conn(0x7f62a80b3050 legacy=0x7f62a806f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a6d9d640 1 -- 172.21.15.156:0/1564829198 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a80b37f0 con 0x7f62a806f7a0 2024-03-29T04:11:01.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.945+0000 7f62a5d9b640 1 -- 172.21.15.156:0/1564829198 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 734664407 0 0) 0x7f6290001110 con 0x7f62a80b3050 2024-03-29T04:11:01.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.946+0000 7f62a6d9d640 1 --2- 172.21.15.156:0/1564829198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a806f7a0 0x7f62a8071750 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009e50 tx=0x7f62a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219fb8c173b6d093 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:01.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.946+0000 7f62a5d9b640 1 -- 172.21.15.156:0/1564829198 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a002ca70 con 0x7f62a806f7a0 2024-03-29T04:11:01.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.946+0000 7f62a5d9b640 1 -- 172.21.15.156:0/1564829198 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a002cc10 con 0x7f62a806f7a0 2024-03-29T04:11:01.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.946+0000 7f62a5d9b640 1 -- 172.21.15.156:0/1564829198 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a00343f0 con 0x7f62a806f7a0 2024-03-29T04:11:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 -- 172.21.15.156:0/1564829198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a806f7a0 msgr2=0x7f62a8071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 --2- 172.21.15.156:0/1564829198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a806f7a0 0x7f62a8071750 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009e50 tx=0x7f62a002bed0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 -- 172.21.15.156:0/1564829198 shutdown_connections 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 --2- 172.21.15.156:0/1564829198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a806f7a0 0x7f62a8071750 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 --2- 172.21.15.156:0/1564829198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a80af6b0 0x7f62a815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 -- 172.21.15.156:0/1564829198 >> 172.21.15.156:0/1564829198 conn(0x7f62a80b1d50 msgr2=0x7f62a80b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 -- 172.21.15.156:0/1564829198 shutdown_connections 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.947+0000 7f62acc14640 1 -- 172.21.15.156:0/1564829198 wait complete. 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.948+0000 7f62acc14640 1 Processor -- start 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.948+0000 7f62acc14640 1 -- start start 2024-03-29T04:11:01.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a806f7a0 0x7f62a814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 0x7f62a81f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a80b3050 0x7f62a8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62a80b4090 con 0x7f62a80af6b0 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62a80b3db0 con 0x7f62a80b3050 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f62a80b45b0 con 0x7f62a806f7a0 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 0x7f62a81f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 0x7f62a81f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35904/0 (socket says 172.21.15.156:35904) 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 -- 172.21.15.156:0/3247244946 learned_addr learned my addr 172.21.15.156:0/3247244946 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:01.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 -- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a806f7a0 msgr2=0x7f62a814e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62a659c640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a806f7a0 0x7f62a814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62a6d9d640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a80b3050 0x7f62a8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a806f7a0 0x7f62a814e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 -- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a80b3050 msgr2=0x7f62a8155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a80b3050 0x7f62a8155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 -- 172.21.15.156:0/3247244946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a814fdc0 con 0x7f62a80af6b0 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629ffff640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 0x7f62a81f0e60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f629000c460 tx=0x7f629000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290018070 con 0x7f62a80af6b0 2024-03-29T04:11:01.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290007500 con 0x7f62a80af6b0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a81f1720 con 0x7f62a80af6b0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.949+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290004a20 con 0x7f62a80af6b0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.950+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a81f1ce0 con 0x7f62a80af6b0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.950+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a8068610 con 0x7f62a80af6b0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.951+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6290004bc0 con 0x7f62a80af6b0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.951+0000 7f629effd640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f628806c530 0x7f628806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.951+0000 7f62a659c640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f628806c530 0x7f628806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:01.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.951+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f629008a780 con 0x7f62a80af6b0 2024-03-29T04:11:01.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.952+0000 7f62a659c640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f628806c530 0x7f628806e9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f62a815bf90 tx=0x7f6298008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:01.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:01.956+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f62a815c7e0 con 0x7f62a80af6b0 2024-03-29T04:11:02.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.126+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62a80b2700 con 0x7f628806c530 2024-03-29T04:11:02.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.126+0000 7f629effd640 1 -- 172.21.15.156:0/3247244946 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13526 (secure 0 0 0) 0x7f62a80b2700 con 0x7f628806c530 2024-03-29T04:11:02.106 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:02.106 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:11:02.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f628806c530 msgr2=0x7f628806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f628806c530 0x7f628806e9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f62a815bf90 tx=0x7f6298008040 comp rx=0 tx=0).stop 2024-03-29T04:11:02.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 msgr2=0x7f62a81f0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 0x7f62a81f0e60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f629000c460 tx=0x7f629000c920 comp rx=0 tx=0).stop 2024-03-29T04:11:02.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 shutdown_connections 2024-03-29T04:11:02.108 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f628806c530 0x7f628806e9f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a80b3050 0x7f62a8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a80af6b0 0x7f62a81f0e60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 --2- 172.21.15.156:0/3247244946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62a806f7a0 0x7f62a814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 >> 172.21.15.156:0/3247244946 conn(0x7f62a80b1d50 msgr2=0x7f62a80b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:02.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 shutdown_connections 2024-03-29T04:11:02.109 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.128+0000 7f62acc14640 1 -- 172.21.15.156:0/3247244946 wait complete. 2024-03-29T04:11:02.119 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-03-29T04:11:01.659933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":552956,"kb_used_data":320168,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754421764,"statfs":{"total":773094113280,"available":772527886336,"internally_reserved":0,"allocated":327852032,"data_stored":325556867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-174309431,"num_objects":-138,"num_object_clones":0,"num_object_copies":-276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-138,"num_whiteouts":0,"num_read":-43,"num_read_kb":-2,"num_write":-332,"num_write_kb":-170316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000913"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022723+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:09:55.022723+0000","last_peered":"2024-03-29T04:09:55.022723+0000","last_clean":"2024-03-29T04:09:55.022723+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:09:55.022723+0000","last_undegraded":"2024-03-29T04:09:55.022723+0000","last_fullsized":"2024-03-29T04:09:55.022723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542174,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73436,"kb_used_data":46420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94298404,"statfs":{"total":96636764160,"available":96561565696,"internally_reserved":0,"allocated":47534080,"data_stored":47246895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52096,"kb_used_data":25080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94319744,"statfs":{"total":96636764160,"available":96583417856,"internally_reserved":0,"allocated":25681920,"data_stored":25422162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542175,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89104,"kb_used_data":62088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94282736,"statfs":{"total":96636764160,"available":96545521664,"internally_reserved":0,"allocated":63578112,"data_stored":63292157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542175,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85748,"kb_used_data":58732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94286092,"statfs":{"total":96636764160,"available":96548958208,"internally_reserved":0,"allocated":60141568,"data_stored":59858595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69504,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":39243776,"data_stored":38912610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31600,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86240,"kb_used_data":55060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94285600,"statfs":{"total":96636764160,"available":96548454400,"internally_reserved":0,"allocated":56381440,"data_stored":56075612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65228,"kb_used_data":34048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306612,"statfs":{"total":96636764160,"available":96569970688,"internally_reserved":0,"allocated":34865152,"data_stored":34581028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:11:02.119 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-29T04:11:02.119 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:11:02.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.240+0000 7f1a9d7fb640 1 Processor -- start 2024-03-29T04:11:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.240+0000 7f1a9d7fb640 1 -- start start 2024-03-29T04:11:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a9d7fb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1a9809aa40 0x7f1a9809ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a9d7fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a980b34b0 con 0x7f1a9815fba0 2024-03-29T04:11:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a9d7fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a980b1d00 con 0x7f1a980b1850 2024-03-29T04:11:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a9d7fb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1a9809b3b0 con 0x7f1a9809aa40 2024-03-29T04:11:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a96ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1a9815fba0 0x7f1a98160a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54672/0 (socket says 172.21.15.156:54672) 2024-03-29T04:11:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a96ffd640 1 -- 172.21.15.156:0/1146397521 learned_addr learned my addr 172.21.15.156:0/1146397521 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.241+0000 7f1a977fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1a9809aa40 0x7f1a9809ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.242+0000 7f1a977fe640 1 -- 172.21.15.156:0/1146397521 >> v1:172.21.15.203:6789/0 conn(0x7f1a980b1850 legacy=0x7f1a980b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.242+0000 7f1a95ffb640 1 -- 172.21.15.156:0/1146397521 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191967388 0 0) 0x7f1a980b34b0 con 0x7f1a9815fba0 2024-03-29T04:11:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.242+0000 7f1a977fe640 1 -- 172.21.15.156:0/1146397521 >> v1:172.21.15.156:6789/0 conn(0x7f1a9815fba0 legacy=0x7f1a98160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.242+0000 7f1a977fe640 1 -- 172.21.15.156:0/1146397521 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a980b1fe0 con 0x7f1a9809aa40 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a977fe640 1 --2- 172.21.15.156:0/1146397521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a9809ae70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097a0 tx=0x7f1a9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a000e36eb66808 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a95ffb640 1 -- 172.21.15.156:0/1146397521 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90035020 con 0x7f1a9809aa40 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a95ffb640 1 -- 172.21.15.156:0/1146397521 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a90031830 con 0x7f1a9809aa40 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a95ffb640 1 -- 172.21.15.156:0/1146397521 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90031c00 con 0x7f1a9809aa40 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/1146397521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 msgr2=0x7f1a9809ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/1146397521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a9809ae70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097a0 tx=0x7f1a9002b8d0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/1146397521 shutdown_connections 2024-03-29T04:11:02.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/1146397521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a9809ae70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/1146397521 >> 172.21.15.156:0/1146397521 conn(0x7f1a980b3d90 msgr2=0x7f1a980b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:02.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/1146397521 shutdown_connections 2024-03-29T04:11:02.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.243+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/1146397521 wait complete. 2024-03-29T04:11:02.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 Processor -- start 2024-03-29T04:11:02.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 -- start start 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a980c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a980b1850 0x7f1a9814f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a9815fba0 0x7f1a981f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1a980b2880 con 0x7f1a9809aa40 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1a980b25a0 con 0x7f1a980b1850 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a9d7fb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1a980b3990 con 0x7f1a9815fba0 2024-03-29T04:11:02.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a96ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a980c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a96ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a980c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35920/0 (socket says 172.21.15.156:35920) 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.245+0000 7f1a96ffd640 1 -- 172.21.15.156:0/3667003835 learned_addr learned my addr 172.21.15.156:0/3667003835 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a977fe640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a9815fba0 0x7f1a981f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a967fc640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a980b1850 0x7f1a9814f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a96ffd640 1 -- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a9815fba0 msgr2=0x7f1a981f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a96ffd640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a9815fba0 0x7f1a981f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a96ffd640 1 -- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a980b1850 msgr2=0x7f1a9814f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a96ffd640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a980b1850 0x7f1a9814f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a96ffd640 1 -- 172.21.15.156:0/3667003835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9814fc50 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a977fe640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a9815fba0 0x7f1a981f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a96ffd640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a980c7ec0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c8c0 tx=0x7f1a8800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88018070 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8800a640 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a980c6e20 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8800aa70 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.246+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a980c73e0 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.247+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a98068d20 con 0x7f1a9809aa40 2024-03-29T04:11:02.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.247+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1a8801b070 con 0x7f1a9809aa40 2024-03-29T04:11:02.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.248+0000 7f1a7f7fe640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a8406c530 0x7f1a8406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.248+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f1a8808aa60 con 0x7f1a9809aa40 2024-03-29T04:11:02.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.248+0000 7f1a967fc640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a8406c530 0x7f1a8406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.248+0000 7f1a967fc640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a8406c530 0x7f1a8406e9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1a9815e3b0 tx=0x7f1a8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:02.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.252+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1a9814fc50 con 0x7f1a9809aa40 2024-03-29T04:11:02.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.423+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a980b0b90 con 0x7f1a8406c530 2024-03-29T04:11:02.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.424+0000 7f1a7f7fe640 1 -- 172.21.15.156:0/3667003835 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13526 (secure 0 0 0) 0x7f1a980b0b90 con 0x7f1a8406c530 2024-03-29T04:11:02.403 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:02.403 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:11:02.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.425+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a8406c530 msgr2=0x7f1a8406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.425+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a8406c530 0x7f1a8406e9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1a9815e3b0 tx=0x7f1a8c009040 comp rx=0 tx=0).stop 2024-03-29T04:11:02.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.425+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 msgr2=0x7f1a980c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.425+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a980c7ec0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c8c0 tx=0x7f1a8800cd80 comp rx=0 tx=0).stop 2024-03-29T04:11:02.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 shutdown_connections 2024-03-29T04:11:02.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1a8406c530 0x7f1a8406e9f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a9815fba0 0x7f1a981f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a980b1850 0x7f1a9814f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 --2- 172.21.15.156:0/3667003835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a9809aa40 0x7f1a980c7ec0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 >> 172.21.15.156:0/3667003835 conn(0x7f1a980b3d90 msgr2=0x7f1a98074530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:02.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 shutdown_connections 2024-03-29T04:11:02.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.426+0000 7f1a9d7fb640 1 -- 172.21.15.156:0/3667003835 wait complete. 2024-03-29T04:11:02.416 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-03-29T04:11:01.659933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":552956,"kb_used_data":320168,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754421764,"statfs":{"total":773094113280,"available":772527886336,"internally_reserved":0,"allocated":327852032,"data_stored":325556867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-174309431,"num_objects":-138,"num_object_clones":0,"num_object_copies":-276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-138,"num_whiteouts":0,"num_read":-43,"num_read_kb":-2,"num_write":-332,"num_write_kb":-170316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000913"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T04:09:55.022723+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:09:55.022723+0000","last_peered":"2024-03-29T04:09:55.022723+0000","last_clean":"2024-03-29T04:09:55.022723+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:09:55.022723+0000","last_undegraded":"2024-03-29T04:09:55.022723+0000","last_fullsized":"2024-03-29T04:09:55.022723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542174,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73436,"kb_used_data":46420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94298404,"statfs":{"total":96636764160,"available":96561565696,"internally_reserved":0,"allocated":47534080,"data_stored":47246895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52096,"kb_used_data":25080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94319744,"statfs":{"total":96636764160,"available":96583417856,"internally_reserved":0,"allocated":25681920,"data_stored":25422162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542175,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89104,"kb_used_data":62088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94282736,"statfs":{"total":96636764160,"available":96545521664,"internally_reserved":0,"allocated":63578112,"data_stored":63292157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542175,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85748,"kb_used_data":58732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94286092,"statfs":{"total":96636764160,"available":96548958208,"internally_reserved":0,"allocated":60141568,"data_stored":59858595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69504,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":39243776,"data_stored":38912610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31600,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86240,"kb_used_data":55060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94285600,"statfs":{"total":96636764160,"available":96548454400,"internally_reserved":0,"allocated":56381440,"data_stored":56075612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65228,"kb_used_data":34048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306612,"statfs":{"total":96636764160,"available":96569970688,"internally_reserved":0,"allocated":34865152,"data_stored":34581028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:11:02.417 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T04:11:02.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.539+0000 7fe53cf8d640 1 Processor -- start 2024-03-29T04:11:02.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.539+0000 7fe53cf8d640 1 -- start start 2024-03-29T04:11:02.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.540+0000 7fe53cf8d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe53815e730 0x7fe5380ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.540+0000 7fe53cf8d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe5380cf3e0 0x7fe5380d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.540+0000 7fe53cf8d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe5380d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.540+0000 7fe53cf8d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe53806c720 con 0x7fe53815e730 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.540+0000 7fe53cf8d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe5380d4c70 con 0x7fe5380cf3e0 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.540+0000 7fe53cf8d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe5380cfa20 con 0x7fe5380d4800 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe53815e730 0x7fe5380ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe53815e730 0x7fe5380ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35922/0 (socket says 172.21.15.156:35922) 2024-03-29T04:11:02.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 -- 172.21.15.156:0/2269475483 learned_addr learned my addr 172.21.15.156:0/2269475483 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe535d9b640 1 --2- 172.21.15.156:0/2269475483 >> v2:172.21.15.203:3300/0 conn(0x7fe5380cf3e0 0x7fe5380d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe536d9d640 1 --2- 172.21.15.156:0/2269475483 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe5380d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 -- 172.21.15.156:0/2269475483 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 msgr2=0x7fe5380d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 --2- 172.21.15.156:0/2269475483 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe5380d0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 -- 172.21.15.156:0/2269475483 >> v2:172.21.15.203:3300/0 conn(0x7fe5380cf3e0 msgr2=0x7fe5380d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 --2- 172.21.15.156:0/2269475483 >> v2:172.21.15.203:3300/0 conn(0x7fe5380cf3e0 0x7fe5380d42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe53659c640 1 -- 172.21.15.156:0/2269475483 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5380cfd00 con 0x7fe53815e730 2024-03-29T04:11:02.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.541+0000 7fe535d9b640 1 --2- 172.21.15.156:0/2269475483 >> v2:172.21.15.203:3300/0 conn(0x7fe5380cf3e0 0x7fe5380d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53659c640 1 --2- 172.21.15.156:0/2269475483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe53815e730 0x7fe5380ceea0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe5380ba4e0 tx=0x7fe53002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfda8ddf6bc9c6cd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53559a640 1 -- 172.21.15.156:0/2269475483 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530035020 con 0x7fe53815e730 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53559a640 1 -- 172.21.15.156:0/2269475483 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe530031870 con 0x7fe53815e730 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53559a640 1 -- 172.21.15.156:0/2269475483 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530031c40 con 0x7fe53815e730 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 -- 172.21.15.156:0/2269475483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe53815e730 msgr2=0x7fe5380ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/2269475483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe53815e730 0x7fe5380ceea0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe5380ba4e0 tx=0x7fe53002b910 comp rx=0 tx=0).stop 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 -- 172.21.15.156:0/2269475483 shutdown_connections 2024-03-29T04:11:02.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/2269475483 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe5380d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/2269475483 >> v2:172.21.15.203:3300/0 conn(0x7fe5380cf3e0 0x7fe5380d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/2269475483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe53815e730 0x7fe5380ceea0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 -- 172.21.15.156:0/2269475483 >> 172.21.15.156:0/2269475483 conn(0x7fe53808bfd0 msgr2=0x7fe53809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:02.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 -- 172.21.15.156:0/2269475483 shutdown_connections 2024-03-29T04:11:02.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.542+0000 7fe53cf8d640 1 -- 172.21.15.156:0/2269475483 wait complete. 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 Processor -- start 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 -- start start 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 0x7fe5381f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe53815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53815e730 0x7fe5380c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5380d2220 con 0x7fe5380cf3e0 2024-03-29T04:11:02.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5380d02c0 con 0x7fe53815e730 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53cf8d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe53815eb10 con 0x7fe5380d4800 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.544+0000 7fe53659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 0x7fe5381f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 0x7fe5381f4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35938/0 (socket says 172.21.15.156:35938) 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 -- 172.21.15.156:0/454974104 learned_addr learned my addr 172.21.15.156:0/454974104 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe535d9b640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe53815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe536d9d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53815e730 0x7fe5380c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 -- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 msgr2=0x7fe53815cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.529 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe53815cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 -- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53815e730 msgr2=0x7fe5380c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53815e730 0x7fe5380c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 -- 172.21.15.156:0/454974104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5380c8570 con 0x7fe5380cf3e0 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53659c640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 0x7fe5381f4020 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe530000c00 tx=0x7fe5300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530035040 con 0x7fe5380cf3e0 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe530045480 con 0x7fe5380cf3e0 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5380c8850 con 0x7fe5380cf3e0 2024-03-29T04:11:02.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530045870 con 0x7fe5380cf3e0 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.545+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5380c8e10 con 0x7fe5380cf3e0 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.546+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5380c9330 con 0x7fe5380cf3e0 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.547+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe530038020 con 0x7fe5380cf3e0 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.547+0000 7fe52effd640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe51806c530 0x7fe51806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.547+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fe5300ad710 con 0x7fe5380cf3e0 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.547+0000 7fe535d9b640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe51806c530 0x7fe51806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:02.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.547+0000 7fe535d9b640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe51806c530 0x7fe51806e9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe53806d6b0 tx=0x7fe520008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:02.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.551+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe53007ba70 con 0x7fe5380cf3e0 2024-03-29T04:11:02.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.725+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe538068610 con 0x7fe5380cf3e0 2024-03-29T04:11:02.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.726+0000 7fe52effd640 1 -- 172.21.15.156:0/454974104 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+13044 (secure 0 0 0) 0x7fe53007fd70 con 0x7fe5380cf3e0 2024-03-29T04:11:02.705 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:02.706 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":37,"fsid":"6a348a99-5402-4006-8e25-31481aaa0bf5","created":"2024-03-29T04:08:29.412260+0000","modified":"2024-03-29T04:10:59.504986+0000","last_up_change":"2024-03-29T04:08:40.748284+0000","last_in_change":"2024-03-29T04:08:31.665343+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T04:08:41.651889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f58abcd-38d8-4959-8eb1-927f56d35ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6809","nonce":1070206615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6811","nonce":1070206615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6815","nonce":1070206615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1070206615},{"type":"v1","addr":"172.21.15.156:6813","nonce":1070206615}]},"public_addr":"172.21.15.156:6809/1070206615","cluster_addr":"172.21.15.156:6811/1070206615","heartbeat_back_addr":"172.21.15.156:6815/1070206615","heartbeat_front_addr":"172.21.15.156:6813/1070206615","state":["exists","up"]},{"osd":1,"uuid":"8eb20861-690f-4730-a707-16db70ce031e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6801","nonce":413252124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6803","nonce":413252124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6807","nonce":413252124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":413252124},{"type":"v1","addr":"172.21.15.156:6805","nonce":413252124}]},"public_addr":"172.21.15.156:6801/413252124","cluster_addr":"172.21.15.156:6803/413252124","heartbeat_back_addr":"172.21.15.156:6807/413252124","heartbeat_front_addr":"172.21.15.156:6805/413252124","state":["exists","up"]},{"osd":2,"uuid":"72345b8a-b884-4048-853a-520cd76b6ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6825","nonce":196737254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6827","nonce":196737254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6831","nonce":196737254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":196737254},{"type":"v1","addr":"172.21.15.156:6829","nonce":196737254}]},"public_addr":"172.21.15.156:6825/196737254","cluster_addr":"172.21.15.156:6827/196737254","heartbeat_back_addr":"172.21.15.156:6831/196737254","heartbeat_front_addr":"172.21.15.156:6829/196737254","state":["exists","up"]},{"osd":3,"uuid":"264ca621-8d00-4dd0-aaac-7bd0f502a666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6817","nonce":3904521534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6819","nonce":3904521534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6823","nonce":3904521534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3904521534},{"type":"v1","addr":"172.21.15.156:6821","nonce":3904521534}]},"public_addr":"172.21.15.156:6817/3904521534","cluster_addr":"172.21.15.156:6819/3904521534","heartbeat_back_addr":"172.21.15.156:6823/3904521534","heartbeat_front_addr":"172.21.15.156:6821/3904521534","state":["exists","up"]},{"osd":4,"uuid":"cb94351b-b0c4-4f09-9ce3-76657401e6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6818","nonce":3017904752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6822","nonce":3017904752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6830","nonce":3017904752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3017904752},{"type":"v1","addr":"172.21.15.203:6826","nonce":3017904752}]},"public_addr":"172.21.15.203:6818/3017904752","cluster_addr":"172.21.15.203:6822/3017904752","heartbeat_back_addr":"172.21.15.203:6830/3017904752","heartbeat_front_addr":"172.21.15.203:6826/3017904752","state":["exists","up"]},{"osd":5,"uuid":"6a37592b-2d57-4865-852b-50824052a319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6809","nonce":2129623277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6811","nonce":2129623277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6815","nonce":2129623277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2129623277},{"type":"v1","addr":"172.21.15.203:6813","nonce":2129623277}]},"public_addr":"172.21.15.203:6809/2129623277","cluster_addr":"172.21.15.203:6811/2129623277","heartbeat_back_addr":"172.21.15.203:6815/2129623277","heartbeat_front_addr":"172.21.15.203:6813/2129623277","state":["exists","up"]},{"osd":6,"uuid":"30c85149-a723-4542-a1f9-e0745f60df69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6801","nonce":505413700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6803","nonce":505413700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6807","nonce":505413700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":505413700},{"type":"v1","addr":"172.21.15.203:6805","nonce":505413700}]},"public_addr":"172.21.15.203:6801/505413700","cluster_addr":"172.21.15.203:6803/505413700","heartbeat_back_addr":"172.21.15.203:6807/505413700","heartbeat_front_addr":"172.21.15.203:6805/505413700","state":["exists","up"]},{"osd":7,"uuid":"f435f10c-bc5c-4df5-a966-310d860eb6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6817","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6819","nonce":3097526350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6821","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6823","nonce":3097526350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6829","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6831","nonce":3097526350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6825","nonce":3097526350},{"type":"v1","addr":"172.21.15.203:6827","nonce":3097526350}]},"public_addr":"172.21.15.203:6819/3097526350","cluster_addr":"172.21.15.203:6823/3097526350","heartbeat_back_addr":"172.21.15.203:6831/3097526350","heartbeat_front_addr":"172.21.15.203:6827/3097526350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.818227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.805296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.799791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.759326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.803958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.888384+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.808507+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T04:08:36.853768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6837/788767406":"2024-03-30T04:10:56.497745+0000","172.21.15.156:6835/788767406":"2024-03-30T04:10:56.497745+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T04:11:02.707 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe51806c530 msgr2=0x7fe51806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe51806c530 0x7fe51806e9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe53806d6b0 tx=0x7fe520008040 comp rx=0 tx=0).stop 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 msgr2=0x7fe5381f4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 0x7fe5381f4020 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe530000c00 tx=0x7fe5300046c0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 shutdown_connections 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe51806c530 0x7fe51806e9f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53815e730 0x7fe5380c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5380d4800 0x7fe53815cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.708 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 --2- 172.21.15.156:0/454974104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5380cf3e0 0x7fe5381f4020 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:02.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 >> 172.21.15.156:0/454974104 conn(0x7fe53808bfd0 msgr2=0x7fe5380b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:02.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 shutdown_connections 2024-03-29T04:11:02.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:02.728+0000 7fe53cf8d640 1 -- 172.21.15.156:0/454974104 wait complete. 2024-03-29T04:11:03.719 INFO:tasks.ceph:Scrubbing osd.0 2024-03-29T04:11:03.720 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:03.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.860+0000 7fd9443bd640 1 Processor -- start 2024-03-29T04:11:03.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.860+0000 7fd9443bd640 1 -- start start 2024-03-29T04:11:03.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd9443bd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c0d0c80 0x7fd93c0d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd9443bd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd93c06c9f0 0x7fd93c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd9443bd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd93c0d21d0 con 0x7fd93c15e670 2024-03-29T04:11:03.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd9443bd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd93c0cff80 con 0x7fd93c06c9f0 2024-03-29T04:11:03.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd9443bd640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd93c0d1620 con 0x7fd93c0d0c80 2024-03-29T04:11:03.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd94295a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd93c06c9f0 0x7fd93c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd94295a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd93c06c9f0 0x7fd93c06ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35954/0 (socket says 172.21.15.156:35954) 2024-03-29T04:11:03.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd942159640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd93c15e670 0x7fd93c0cf4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54680/0 (socket says 172.21.15.156:54680) 2024-03-29T04:11:03.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd94295a640 1 -- 172.21.15.156:0/2716080246 learned_addr learned my addr 172.21.15.156:0/2716080246 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:03.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.861+0000 7fd941958640 1 --2- 172.21.15.156:0/2716080246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c0d0c80 0x7fd93c0d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.862+0000 7fd94295a640 1 -- 172.21.15.156:0/2716080246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c0d0c80 msgr2=0x7fd93c0d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:03.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.862+0000 7fd94295a640 1 --2- 172.21.15.156:0/2716080246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c0d0c80 0x7fd93c0d10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.862+0000 7fd94295a640 1 -- 172.21.15.156:0/2716080246 >> v1:172.21.15.156:6789/0 conn(0x7fd93c15e670 legacy=0x7fd93c0cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.862+0000 7fd94295a640 1 -- 172.21.15.156:0/2716080246 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c0d2470 con 0x7fd93c06c9f0 2024-03-29T04:11:03.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.862+0000 7fd941157640 1 -- 172.21.15.156:0/2716080246 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885015168 0 0) 0x7fd93c0d21d0 con 0x7fd93c15e670 2024-03-29T04:11:03.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.862+0000 7fd94295a640 1 --2- 172.21.15.156:0/2716080246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c06ce80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd934009e50 tx=0x7fd93402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94894a7bb0cf5fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:03.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd941157640 1 -- 172.21.15.156:0/2716080246 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93402cb00 con 0x7fd93c06c9f0 2024-03-29T04:11:03.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd941157640 1 -- 172.21.15.156:0/2716080246 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd93402cca0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd941157640 1 -- 172.21.15.156:0/2716080246 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934034610 con 0x7fd93c06c9f0 2024-03-29T04:11:03.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 -- 172.21.15.156:0/2716080246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 msgr2=0x7fd93c06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 --2- 172.21.15.156:0/2716080246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c06ce80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd934009e50 tx=0x7fd93402bf60 comp rx=0 tx=0).stop 2024-03-29T04:11:03.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 -- 172.21.15.156:0/2716080246 shutdown_connections 2024-03-29T04:11:03.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 --2- 172.21.15.156:0/2716080246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c06ce80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 --2- 172.21.15.156:0/2716080246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c0d0c80 0x7fd93c0d10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 -- 172.21.15.156:0/2716080246 >> 172.21.15.156:0/2716080246 conn(0x7fd93c08bfd0 msgr2=0x7fd93c0d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:03.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 -- 172.21.15.156:0/2716080246 shutdown_connections 2024-03-29T04:11:03.847 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.863+0000 7fd9443bd640 1 -- 172.21.15.156:0/2716080246 wait complete. 2024-03-29T04:11:03.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 Processor -- start 2024-03-29T04:11:03.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 -- start start 2024-03-29T04:11:03.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd93c0d0c80 0x7fd93c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c15e670 0x7fd93c09f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c15ea50 con 0x7fd93c06c9f0 2024-03-29T04:11:03.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c06dd90 con 0x7fd93c0d0c80 2024-03-29T04:11:03.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd9443bd640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fd93c0cfbd0 con 0x7fd93c15e670 2024-03-29T04:11:03.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd942159640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.865+0000 7fd942159640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c15cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35970/0 (socket says 172.21.15.156:35970) 2024-03-29T04:11:03.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 -- 172.21.15.156:0/3882479932 learned_addr learned my addr 172.21.15.156:0/3882479932 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:03.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd94295a640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c15e670 0x7fd93c09f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd941958640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd93c0d0c80 0x7fd93c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 -- 172.21.15.156:0/3882479932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c15e670 msgr2=0x7fd93c09f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd93c15e670 0x7fd93c09f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 -- 172.21.15.156:0/3882479932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd93c0d0c80 msgr2=0x7fd93c15ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd93c0d0c80 0x7fd93c15ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 -- 172.21.15.156:0/3882479932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c0b81d0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd942159640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c15cbe0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c430 tx=0x7fd92c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:03.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0074e0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92c0047e0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c004bd0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93c0b84b0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.866+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd93c0b8a70 con 0x7fd93c06c9f0 2024-03-29T04:11:03.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.867+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd93c0a0210 con 0x7fd93c06c9f0 2024-03-29T04:11:03.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd92c011070 con 0x7fd93c06c9f0 2024-03-29T04:11:03.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd932ffd640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd92006c530 0x7fd92006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fd92c08b7d0 con 0x7fd93c06c9f0 2024-03-29T04:11:03.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd941958640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd92006c530 0x7fd92006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd932ffd640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fd9200723f0 0x7fd920074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:03.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 --> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd92c01ee50 con 0x7fd9200723f0 2024-03-29T04:11:03.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd92c08bb80 con 0x7fd93c06c9f0 2024-03-29T04:11:03.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.868+0000 7fd94295a640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fd9200723f0 0x7fd920074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:03.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.869+0000 7fd941958640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd92006c530 0x7fd92006e9f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd93c15d7c0 tx=0x7fd938008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:03.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.869+0000 7fd94295a640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fd9200723f0 0x7fd920074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:03.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.869+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== osd.0 v2:172.21.15.156:6808/1070206615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd92c01ee50 con 0x7fd9200723f0 2024-03-29T04:11:03.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 --> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd93c068610 con 0x7fd9200723f0 2024-03-29T04:11:03.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd932ffd640 1 -- 172.21.15.156:0/3882479932 <== osd.0 v2:172.21.15.156:6808/1070206615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd93c068610 con 0x7fd9200723f0 2024-03-29T04:11:03.864 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:03.864 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:03.864 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:03.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fd9200723f0 msgr2=0x7fd920074810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6808/1070206615,v1:172.21.15.156:6809/1070206615] conn(0x7fd9200723f0 0x7fd920074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd92006c530 msgr2=0x7fd92006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fd92006c530 0x7fd92006e9f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd93c15d7c0 tx=0x7fd938008040 comp rx=0 tx=0).stop 2024-03-29T04:11:03.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 msgr2=0x7fd93c15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:03.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 --2- 172.21.15.156:0/3882479932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd93c06c9f0 0x7fd93c15cbe0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c430 tx=0x7fd92c00c8f0 comp rx=0 tx=0).stop 2024-03-29T04:11:03.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd94295a640 1 -- 172.21.15.156:0/3882479932 reap_dead start 2024-03-29T04:11:03.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 shutdown_connections 2024-03-29T04:11:03.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 >> 172.21.15.156:0/3882479932 conn(0x7fd93c08bfd0 msgr2=0x7fd93c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:03.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 shutdown_connections 2024-03-29T04:11:03.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:03.883+0000 7fd9443bd640 1 -- 172.21.15.156:0/3882479932 wait complete. 2024-03-29T04:11:03.875 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-29T04:11:03.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.000+0000 7fc248d0e640 1 Processor -- start 2024-03-29T04:11:04.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.000+0000 7fc248d0e640 1 -- start start 2024-03-29T04:11:04.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc248d0e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc2441415e0 0x7fc244144460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc248d0e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc2441449a0 0x7fc2440b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc248d0e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc244143ec0 0x7fc2440b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc248d0e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc244141d10 con 0x7fc244143ec0 2024-03-29T04:11:04.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc248d0e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc244141f60 con 0x7fc2441415e0 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc248d0e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc244142240 con 0x7fc2441449a0 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc244143ec0 0x7fc2440b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc244143ec0 0x7fc2440b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35982/0 (socket says 172.21.15.156:35982) 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc24259c640 1 --2- 172.21.15.156:0/2236145144 >> v2:172.21.15.203:3300/0 conn(0x7fc2441415e0 0x7fc244144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 -- 172.21.15.156:0/2236145144 learned_addr learned my addr 172.21.15.156:0/2236145144 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc241d9b640 1 --2- 172.21.15.156:0/2236145144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc2441449a0 0x7fc2440b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 -- 172.21.15.156:0/2236145144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc2441449a0 msgr2=0x7fc2440b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:04.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 --2- 172.21.15.156:0/2236145144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc2441449a0 0x7fc2440b54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 -- 172.21.15.156:0/2236145144 >> v2:172.21.15.203:3300/0 conn(0x7fc2441415e0 msgr2=0x7fc244144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:04.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 --2- 172.21.15.156:0/2236145144 >> v2:172.21.15.203:3300/0 conn(0x7fc2441415e0 0x7fc244144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc242d9d640 1 -- 172.21.15.156:0/2236145144 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc244142520 con 0x7fc244143ec0 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.001+0000 7fc24259c640 1 --2- 172.21.15.156:0/2236145144 >> v2:172.21.15.203:3300/0 conn(0x7fc2441415e0 0x7fc244144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.002+0000 7fc242d9d640 1 --2- 172.21.15.156:0/2236145144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc244143ec0 0x7fc2440b5a00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc244142450 tx=0x7fc23c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9234f5055ce8dfd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.002+0000 7fc24159a640 1 -- 172.21.15.156:0/2236145144 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c02c9e0 con 0x7fc244143ec0 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.002+0000 7fc24159a640 1 -- 172.21.15.156:0/2236145144 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c02cb80 con 0x7fc244143ec0 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.002+0000 7fc24159a640 1 -- 172.21.15.156:0/2236145144 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0343f0 con 0x7fc244143ec0 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 -- 172.21.15.156:0/2236145144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc244143ec0 msgr2=0x7fc2440b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 --2- 172.21.15.156:0/2236145144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc244143ec0 0x7fc2440b5a00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc244142450 tx=0x7fc23c02be40 comp rx=0 tx=0).stop 2024-03-29T04:11:04.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 -- 172.21.15.156:0/2236145144 shutdown_connections 2024-03-29T04:11:04.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 --2- 172.21.15.156:0/2236145144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc244143ec0 0x7fc2440b5a00 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 --2- 172.21.15.156:0/2236145144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc2441449a0 0x7fc2440b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 --2- 172.21.15.156:0/2236145144 >> v2:172.21.15.203:3300/0 conn(0x7fc2441415e0 0x7fc244144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 -- 172.21.15.156:0/2236145144 >> 172.21.15.156:0/2236145144 conn(0x7fc2440b3eb0 msgr2=0x7fc2440b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 -- 172.21.15.156:0/2236145144 shutdown_connections 2024-03-29T04:11:04.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.003+0000 7fc248d0e640 1 -- 172.21.15.156:0/2236145144 wait complete. 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.004+0000 7fc248d0e640 1 Processor -- start 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.004+0000 7fc248d0e640 1 -- start start 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc248d0e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 0x7fc24413cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc248d0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2441449a0 0x7fc244144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc248d0e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc244145360 0x7fc2441468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc248d0e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc24406c6c0 con 0x7fc2441415e0 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc248d0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441419c0 con 0x7fc2441449a0 2024-03-29T04:11:04.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc248d0e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fc24406c840 con 0x7fc244145360 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 0x7fc24413cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 0x7fc24413cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35986/0 (socket says 172.21.15.156:35986) 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 -- 172.21.15.156:0/3929913917 learned_addr learned my addr 172.21.15.156:0/3929913917 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc241d9b640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2441449a0 0x7fc244144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 -- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc244145360 msgr2=0x7fc2441468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc244145360 0x7fc2441468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 -- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2441449a0 msgr2=0x7fc244144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2441449a0 0x7fc244144e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.005+0000 7fc24259c640 1 -- 172.21.15.156:0/3929913917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc244145800 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.006+0000 7fc24259c640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 0x7fc24413cd10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc23400e960 tx=0x7fc23400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.006+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234017070 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.006+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23400ca00 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.006+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc244145ae0 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.006+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23400cdd0 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.006+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24413d4c0 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.007+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc244070e10 con 0x7fc2441415e0 2024-03-29T04:11:04.072 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.008+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc23401a070 con 0x7fc2441415e0 2024-03-29T04:11:04.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.008+0000 7fc232ffd640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22406c530 0x7fc22406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.008+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fc23408b580 con 0x7fc2441415e0 2024-03-29T04:11:04.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.008+0000 7fc241d9b640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22406c530 0x7fc22406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.009+0000 7fc241d9b640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22406c530 0x7fc22406e9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc244147f20 tx=0x7fc238008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.073 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.013+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc234059f30 con 0x7fc2441415e0 2024-03-29T04:11:04.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.195+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fc24404f9d0 con 0x7fc22406c530 2024-03-29T04:11:04.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.195+0000 7fc232ffd640 1 -- 172.21.15.156:0/3929913917 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc24404f9d0 con 0x7fc22406c530 2024-03-29T04:11:04.174 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 0 to deep-scrub 2024-03-29T04:11:04.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22406c530 msgr2=0x7fc22406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22406c530 0x7fc22406e9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc244147f20 tx=0x7fc238008040 comp rx=0 tx=0).stop 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 msgr2=0x7fc24413cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 0x7fc24413cd10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc23400e960 tx=0x7fc23400ee20 comp rx=0 tx=0).stop 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 shutdown_connections 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fc22406c530 0x7fc22406e9f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc244145360 0x7fc2441468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2441449a0 0x7fc244144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 --2- 172.21.15.156:0/3929913917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2441415e0 0x7fc24413cd10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 >> 172.21.15.156:0/3929913917 conn(0x7fc2440b3eb0 msgr2=0x7fc2440b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 shutdown_connections 2024-03-29T04:11:04.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.197+0000 7fc248d0e640 1 -- 172.21.15.156:0/3929913917 wait complete. 2024-03-29T04:11:04.187 INFO:tasks.ceph:Scrubbing osd.1 2024-03-29T04:11:04.188 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:04.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 Processor -- start 2024-03-29T04:11:04.295 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 -- start start 2024-03-29T04:11:04.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc15ab50 0x7f1cbc15f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1cbc0b4d50 0x7f1cbc0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cbc06d9c0 con 0x7f1cbc0ad7a0 2024-03-29T04:11:04.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1cbc15af30 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.316+0000 7f1cc270e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc06dca0 con 0x7f1cbc15ab50 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1cbc0b4d50 0x7f1cbc0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1cbc0b4d50 0x7f1cbc0ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36002/0 (socket says 172.21.15.156:36002) 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1796873089 learned_addr learned my addr 172.21.15.156:0/1796873089 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbbfff640 1 --2- 172.21.15.156:0/1796873089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc15ab50 0x7f1cbc15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1796873089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc15ab50 msgr2=0x7f1cbc15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 --2- 172.21.15.156:0/1796873089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc15ab50 0x7f1cbc15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1796873089 >> v1:172.21.15.203:6789/0 conn(0x7f1cbc0ad7a0 legacy=0x7f1cbc072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbaffd640 1 -- 172.21.15.156:0/1796873089 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041544864 0 0) 0x7f1cbc06d9c0 con 0x7f1cbc0ad7a0 2024-03-29T04:11:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.317+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1796873089 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc06df80 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.318+0000 7f1cbb7fe640 1 --2- 172.21.15.156:0/1796873089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc0ad260 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000bb40 tx=0x7f1cb002e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e9cd05ef8115e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.318+0000 7f1cbaffd640 1 -- 172.21.15.156:0/1796873089 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb003b070 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.318+0000 7f1cbaffd640 1 -- 172.21.15.156:0/1796873089 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb0036440 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.318+0000 7f1cbaffd640 1 -- 172.21.15.156:0/1796873089 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0036810 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 -- 172.21.15.156:0/1796873089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 msgr2=0x7f1cbc0ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 --2- 172.21.15.156:0/1796873089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc0ad260 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000bb40 tx=0x7f1cb002e1f0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 -- 172.21.15.156:0/1796873089 shutdown_connections 2024-03-29T04:11:04.302 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 --2- 172.21.15.156:0/1796873089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc0ad260 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 --2- 172.21.15.156:0/1796873089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc15ab50 0x7f1cbc15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 -- 172.21.15.156:0/1796873089 >> 172.21.15.156:0/1796873089 conn(0x7f1cbc0b21e0 msgr2=0x7f1cbc0b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 -- 172.21.15.156:0/1796873089 shutdown_connections 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.319+0000 7f1cc270e640 1 -- 172.21.15.156:0/1796873089 wait complete. 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.320+0000 7f1cc270e640 1 Processor -- start 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.320+0000 7f1cc270e640 1 -- start start 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc270e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc0ad7a0 0x7f1cbc1487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.303 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc270e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc270e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cbc15ab50 0x7f1cbc159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc270e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc072870 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc270e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc0b3510 con 0x7f1cbc15ab50 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc270e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc0729f0 con 0x7f1cbc0ad7a0 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36014/0 (socket says 172.21.15.156:36014) 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1817241417 learned_addr learned my addr 172.21.15.156:0/1817241417 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbbfff640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc0ad7a0 0x7f1cbc1487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cc0cab640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cbc15ab50 0x7f1cbc159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1817241417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc0ad7a0 msgr2=0x7f1cbc1487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cbc0ad7a0 0x7f1cbc1487e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1817241417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cbc15ab50 msgr2=0x7f1cbc159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cbc15ab50 0x7f1cbc159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 -- 172.21.15.156:0/1817241417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc145f80 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.321+0000 7f1cbb7fe640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc159890 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000b2c0 tx=0x7f1cb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.322+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb003b040 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.322+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb0004230 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.322+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cbc0cd780 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.322+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0003610 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.322+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cbc0cdd40 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.323+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1cbc0ce120 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.323+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1cb00037b0 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cb8ff9640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1ca806c530 0x7f1ca806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f1cb00b0190 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cbbfff640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1ca806c530 0x7f1ca806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cb8ff9640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f1ca80723f0 0x7f1ca8074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 --> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cb0036440 con 0x7f1ca80723f0 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1cb00b7050 con 0x7f1cbc0b4d50 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cc0cab640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f1ca80723f0 0x7f1ca8074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cbbfff640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1ca806c530 0x7f1ca806e9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009aa0 tx=0x7f1cac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.324+0000 7f1cc0cab640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f1ca80723f0 0x7f1ca8074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.325+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== osd.1 v2:172.21.15.156:6800/413252124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cb0036440 con 0x7f1ca80723f0 2024-03-29T04:11:04.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.338+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 --> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1cbc068610 con 0x7f1ca80723f0 2024-03-29T04:11:04.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cb8ff9640 1 -- 172.21.15.156:0/1817241417 <== osd.1 v2:172.21.15.156:6800/413252124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1cbc068610 con 0x7f1ca80723f0 2024-03-29T04:11:04.318 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:04.318 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:04.318 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:04.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f1ca80723f0 msgr2=0x7f1ca8074810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6800/413252124,v1:172.21.15.156:6801/413252124] conn(0x7f1ca80723f0 0x7f1ca8074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1ca806c530 msgr2=0x7f1ca806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1ca806c530 0x7f1ca806e9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009aa0 tx=0x7f1cac009190 comp rx=0 tx=0).stop 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 msgr2=0x7f1cbc159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 --2- 172.21.15.156:0/1817241417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cbc0b4d50 0x7f1cbc159890 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000b2c0 tx=0x7f1cb00046c0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc0cab640 1 -- 172.21.15.156:0/1817241417 reap_dead start 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 shutdown_connections 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 >> 172.21.15.156:0/1817241417 conn(0x7f1cbc0b21e0 msgr2=0x7f1cbc06e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 shutdown_connections 2024-03-29T04:11:04.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.339+0000 7f1cc270e640 1 -- 172.21.15.156:0/1817241417 wait complete. 2024-03-29T04:11:04.329 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-29T04:11:04.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.453+0000 7f51aae9a640 1 Processor -- start 2024-03-29T04:11:04.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.454+0000 7f51aae9a640 1 -- start start 2024-03-29T04:11:04.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.454+0000 7f51aae9a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a4145c70 0x7f51a40b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.454+0000 7f51aae9a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f51a40b4310 0x7f51a40cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.454+0000 7f51aae9a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a415cb10 con 0x7f51a40cf340 2024-03-29T04:11:04.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.454+0000 7f51aae9a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f51a40b4770 con 0x7f51a40b4310 2024-03-29T04:11:04.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.454+0000 7f51aae9a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f51a415cdf0 con 0x7f51a4145c70 2024-03-29T04:11:04.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f51a40b4310 0x7f51a40cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a9437640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f51a40cf340 0x7f51a406f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54688/0 (socket says 172.21.15.156:54688) 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a9437640 1 -- 172.21.15.156:0/868822843 learned_addr learned my addr 172.21.15.156:0/868822843 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a8c36640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a4145c70 0x7f51a40b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a3fff640 1 -- 172.21.15.156:0/868822843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a4145c70 msgr2=0x7f51a40b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a3fff640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a4145c70 0x7f51a40b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a3fff640 1 -- 172.21.15.156:0/868822843 >> v1:172.21.15.156:6789/0 conn(0x7f51a40cf340 legacy=0x7f51a406f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a37fe640 1 -- 172.21.15.156:0/868822843 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3753238363 0 0) 0x7f51a415cb10 con 0x7f51a40cf340 2024-03-29T04:11:04.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a3fff640 1 -- 172.21.15.156:0/868822843 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a415d0d0 con 0x7f51a40b4310 2024-03-29T04:11:04.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.455+0000 7f51a8c36640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a4145c70 0x7f51a40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:04.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51a3fff640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a40b4310 0x7f51a40cee00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f51900097a0 tx=0x7f519002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372fa2f2d4282f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51a37fe640 1 -- 172.21.15.156:0/868822843 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190031780 con 0x7f51a40b4310 2024-03-29T04:11:04.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51a37fe640 1 -- 172.21.15.156:0/868822843 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5190037070 con 0x7f51a40b4310 2024-03-29T04:11:04.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51a37fe640 1 -- 172.21.15.156:0/868822843 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190031b40 con 0x7f51a40b4310 2024-03-29T04:11:04.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 -- 172.21.15.156:0/868822843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a40b4310 msgr2=0x7f51a40cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a40b4310 0x7f51a40cee00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f51900097a0 tx=0x7f519002b820 comp rx=0 tx=0).stop 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 -- 172.21.15.156:0/868822843 shutdown_connections 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a40b4310 0x7f51a40cee00 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 --2- 172.21.15.156:0/868822843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a4145c70 0x7f51a40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 -- 172.21.15.156:0/868822843 >> 172.21.15.156:0/868822843 conn(0x7f51a40b7660 msgr2=0x7f51a40b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 -- 172.21.15.156:0/868822843 shutdown_connections 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.456+0000 7f51aae9a640 1 -- 172.21.15.156:0/868822843 wait complete. 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.457+0000 7f51aae9a640 1 Processor -- start 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 -- start start 2024-03-29T04:11:04.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a40b4310 0x7f51a413e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a40cf340 0x7f51a40b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 0x7f51a413dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f51a40cf930 con 0x7f51a4145c70 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51a40cf7b0 con 0x7f51a40b4310 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51aae9a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f51a4146050 con 0x7f51a40cf340 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.458+0000 7f51a9437640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 0x7f51a413dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 0x7f51a413dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36040/0 (socket says 172.21.15.156:36040) 2024-03-29T04:11:04.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 -- 172.21.15.156:0/1415978878 learned_addr learned my addr 172.21.15.156:0/1415978878 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a3fff640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a40cf340 0x7f51a40b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a8c36640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a40b4310 0x7f51a413e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 -- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a40cf340 msgr2=0x7f51a40b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a40cf340 0x7f51a40b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 -- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a40b4310 msgr2=0x7f51a413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a40b4310 0x7f51a413e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 -- 172.21.15.156:0/1415978878 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a413e2e0 con 0x7f51a4145c70 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a8c36640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a40b4310 0x7f51a413e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:04.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a3fff640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a40cf340 0x7f51a40b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a9437640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 0x7f51a413dda0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f519c00c7a0 tx=0x7f519c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c018070 con 0x7f51a4145c70 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f519c00a640 con 0x7f51a4145c70 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a4141860 con 0x7f51a4145c70 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c00aa70 con 0x7f51a4145c70 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.459+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a4141e20 con 0x7f51a4145c70 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.460+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a404f9d0 con 0x7f51a4145c70 2024-03-29T04:11:04.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.461+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f519c01b070 con 0x7f51a4145c70 2024-03-29T04:11:04.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.461+0000 7f51a17fa640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f518806c530 0x7f518806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.461+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f519c08aa40 con 0x7f51a4145c70 2024-03-29T04:11:04.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.461+0000 7f51a8c36640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f518806c530 0x7f518806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.461+0000 7f51a8c36640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f518806c530 0x7f518806e9f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f51a40b6540 tx=0x7f5198008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.466+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f519c0592c0 con 0x7f51a4145c70 2024-03-29T04:11:04.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.764+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f51a415be60 con 0x7f518806c530 2024-03-29T04:11:04.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.764+0000 7f51a17fa640 1 -- 172.21.15.156:0/1415978878 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f51a415be60 con 0x7f518806c530 2024-03-29T04:11:04.744 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 1 to deep-scrub 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f518806c530 msgr2=0x7f518806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f518806c530 0x7f518806e9f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f51a40b6540 tx=0x7f5198008040 comp rx=0 tx=0).stop 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 msgr2=0x7f51a413dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 0x7f51a413dda0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f519c00c7a0 tx=0x7f519c00cc60 comp rx=0 tx=0).stop 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 shutdown_connections 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f518806c530 0x7f518806e9f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a4145c70 0x7f51a413dda0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f51a40cf340 0x7f51a40b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 --2- 172.21.15.156:0/1415978878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a40b4310 0x7f51a413e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 >> 172.21.15.156:0/1415978878 conn(0x7f51a40b7660 msgr2=0x7f51a4143a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 shutdown_connections 2024-03-29T04:11:04.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.766+0000 7f51aae9a640 1 -- 172.21.15.156:0/1415978878 wait complete. 2024-03-29T04:11:04.757 INFO:tasks.ceph:Scrubbing osd.2 2024-03-29T04:11:04.757 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:04.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.884+0000 7f9f9b863640 1 Processor -- start 2024-03-29T04:11:04.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.885+0000 7f9f9b863640 1 -- start start 2024-03-29T04:11:04.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f9b863640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f9b863640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9f940cfe60 0x7f9f94160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.866 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f9b863640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9f941605b0 0x7f9f9415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f9b863640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9f940d0f70 con 0x7f9f941605b0 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f9b863640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9f940d0360 con 0x7f9f940cfe60 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f9b863640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9f940d0640 con 0x7f9f940b7f50 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9f941605b0 0x7f9f9415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9f941605b0 0x7f9f9415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36044/0 (socket says 172.21.15.156:36044) 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 -- 172.21.15.156:0/3526665257 learned_addr learned my addr 172.21.15.156:0/3526665257 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f98dfe640 1 --2- 172.21.15.156:0/3526665257 >> v2:172.21.15.203:3300/0 conn(0x7f9f940cfe60 0x7f9f94160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f995ff640 1 --2- 172.21.15.156:0/3526665257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 -- 172.21.15.156:0/3526665257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 msgr2=0x7f9f940cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 --2- 172.21.15.156:0/3526665257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 -- 172.21.15.156:0/3526665257 >> v2:172.21.15.203:3300/0 conn(0x7f9f940cfe60 msgr2=0x7f9f94160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 --2- 172.21.15.156:0/3526665257 >> v2:172.21.15.203:3300/0 conn(0x7f9f940cfe60 0x7f9f94160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f99e00640 1 -- 172.21.15.156:0/3526665257 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f940d0920 con 0x7f9f941605b0 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.886+0000 7f9f98dfe640 1 --2- 172.21.15.156:0/3526665257 >> v2:172.21.15.203:3300/0 conn(0x7f9f940cfe60 0x7f9f94160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:04.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.887+0000 7f9f99e00640 1 --2- 172.21.15.156:0/3526665257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9415bee0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406c590 tx=0x7f9f8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703de9fc20a35238 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.887+0000 7f9f8bfff640 1 -- 172.21.15.156:0/3526665257 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c02ca70 con 0x7f9f941605b0 2024-03-29T04:11:04.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.887+0000 7f9f8bfff640 1 -- 172.21.15.156:0/3526665257 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c02cc10 con 0x7f9f941605b0 2024-03-29T04:11:04.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.887+0000 7f9f8bfff640 1 -- 172.21.15.156:0/3526665257 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0343f0 con 0x7f9f941605b0 2024-03-29T04:11:04.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 -- 172.21.15.156:0/3526665257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 msgr2=0x7f9f9415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3526665257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9415bee0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406c590 tx=0x7f9f8c02bed0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 -- 172.21.15.156:0/3526665257 shutdown_connections 2024-03-29T04:11:04.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3526665257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9415bee0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3526665257 >> v2:172.21.15.203:3300/0 conn(0x7f9f940cfe60 0x7f9f94160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3526665257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 -- 172.21.15.156:0/3526665257 >> 172.21.15.156:0/3526665257 conn(0x7f9f940d2290 msgr2=0x7f9f940d26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 -- 172.21.15.156:0/3526665257 shutdown_connections 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.888+0000 7f9f9b863640 1 -- 172.21.15.156:0/3526665257 wait complete. 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.889+0000 7f9f9b863640 1 Processor -- start 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.889+0000 7f9f9b863640 1 -- start start 2024-03-29T04:11:04.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f9b863640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f9b863640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f940cfe60 0x7f9f9414a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f9b863640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9414a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f9b863640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9f940cd1b0 con 0x7f9f941605b0 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f9b863640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f940cd030 con 0x7f9f940cfe60 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f9b863640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9415c420 con 0x7f9f940b7f50 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9414a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9414a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36050/0 (socket says 172.21.15.156:36050) 2024-03-29T04:11:04.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 -- 172.21.15.156:0/3794804130 learned_addr learned my addr 172.21.15.156:0/3794804130 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f995ff640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f98dfe640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f940cfe60 0x7f9f9414a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 -- 172.21.15.156:0/3794804130 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 msgr2=0x7f9f940c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9f940b7f50 0x7f9f940c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 -- 172.21.15.156:0/3794804130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f940cfe60 msgr2=0x7f9f9414a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f940cfe60 0x7f9f9414a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.890+0000 7f9f99e00640 1 -- 172.21.15.156:0/3794804130 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f940ca840 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.891+0000 7f9f99e00640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9414a7e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c036040 tx=0x7f9f8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.891+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c044020 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.891+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c040690 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.891+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f940c6ac0 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.891+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c040a80 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.891+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f940c7080 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.892+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f94068610 con 0x7f9f941605b0 2024-03-29T04:11:04.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.892+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9f8c047020 con 0x7f9f941605b0 2024-03-29T04:11:04.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f89ffb640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9f7806c530 0x7f9f7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f9f8c0ade80 con 0x7f9f941605b0 2024-03-29T04:11:04.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f995ff640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9f7806c530 0x7f9f7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f89ffb640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f9f780723f0 0x7f9f78074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:04.878 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 --> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f78074ec0 con 0x7f9f780723f0 2024-03-29T04:11:04.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9f8c0b0580 con 0x7f9f941605b0 2024-03-29T04:11:04.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f98dfe640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f9f780723f0 0x7f9f78074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:04.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f995ff640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9f7806c530 0x7f9f7806e9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000aac0 tx=0x7f9f90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.893+0000 7f9f98dfe640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f9f780723f0 0x7f9f78074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:04.879 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.894+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== osd.2 v2:172.21.15.156:6824/196737254 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f78074ec0 con 0x7f9f780723f0 2024-03-29T04:11:04.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.907+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 --> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9f940c8a70 con 0x7f9f780723f0 2024-03-29T04:11:04.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.907+0000 7f9f89ffb640 1 -- 172.21.15.156:0/3794804130 <== osd.2 v2:172.21.15.156:6824/196737254 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9f940c8a70 con 0x7f9f780723f0 2024-03-29T04:11:04.887 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:04.887 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:04.887 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:04.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.907+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f9f780723f0 msgr2=0x7f9f78074810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.907+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6824/196737254,v1:172.21.15.156:6825/196737254] conn(0x7f9f780723f0 0x7f9f78074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:04.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9f7806c530 msgr2=0x7f9f7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f9f7806c530 0x7f9f7806e9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000aac0 tx=0x7f9f90008040 comp rx=0 tx=0).stop 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 msgr2=0x7f9f9414a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 --2- 172.21.15.156:0/3794804130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f941605b0 0x7f9f9414a7e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c036040 tx=0x7f9f8c004a40 comp rx=0 tx=0).stop 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f99e00640 1 -- 172.21.15.156:0/3794804130 reap_dead start 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 shutdown_connections 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 >> 172.21.15.156:0/3794804130 conn(0x7f9f940d2290 msgr2=0x7f9f940b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 shutdown_connections 2024-03-29T04:11:04.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:04.908+0000 7f9f9b863640 1 -- 172.21.15.156:0/3794804130 wait complete. 2024-03-29T04:11:04.899 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-29T04:11:05.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.022+0000 7fa94943c640 1 Processor -- start 2024-03-29T04:11:05.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.022+0000 7fa94943c640 1 -- start start 2024-03-29T04:11:05.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa94943c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa94415cb50 0x7fa94415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa94943c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa94409ba50 0x7fa944160760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa94943c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94415d470 con 0x7fa9440cd940 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa94943c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa9440d01e0 con 0x7fa94409ba50 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa94943c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa9440d04c0 con 0x7fa94415cb50 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa9437fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa94409ba50 0x7fa944160760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa942ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa94415cb50 0x7fa94415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa9437fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa94409ba50 0x7fa944160760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36066/0 (socket says 172.21.15.156:36066) 2024-03-29T04:11:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa9427fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa9440cd940 0x7fa944160050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:39072/0 (socket says 172.21.15.156:39072) 2024-03-29T04:11:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.023+0000 7fa9437fe640 1 -- 172.21.15.156:0/2634103498 learned_addr learned my addr 172.21.15.156:0/2634103498 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.024+0000 7fa9437fe640 1 -- 172.21.15.156:0/2634103498 >> v2:172.21.15.203:3300/0 conn(0x7fa94415cb50 msgr2=0x7fa94415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.024+0000 7fa9437fe640 1 --2- 172.21.15.156:0/2634103498 >> v2:172.21.15.203:3300/0 conn(0x7fa94415cb50 0x7fa94415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.024+0000 7fa9437fe640 1 -- 172.21.15.156:0/2634103498 >> v1:172.21.15.203:6789/0 conn(0x7fa9440cd940 legacy=0x7fa944160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.024+0000 7fa9437fe640 1 -- 172.21.15.156:0/2634103498 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9440d07a0 con 0x7fa94409ba50 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.025+0000 7fa9437fe640 1 --2- 172.21.15.156:0/2634103498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa94409ba50 0x7fa944160760 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa94406f410 tx=0x7fa93c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36613739779027 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.025+0000 7fa941ffb640 1 -- 172.21.15.156:0/2634103498 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c02cb20 con 0x7fa94409ba50 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.025+0000 7fa941ffb640 1 -- 172.21.15.156:0/2634103498 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa93c02ccc0 con 0x7fa94409ba50 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.025+0000 7fa941ffb640 1 -- 172.21.15.156:0/2634103498 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c034610 con 0x7fa94409ba50 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 -- 172.21.15.156:0/2634103498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa94409ba50 msgr2=0x7fa944160760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 --2- 172.21.15.156:0/2634103498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa94409ba50 0x7fa944160760 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa94406f410 tx=0x7fa93c02bf80 comp rx=0 tx=0).stop 2024-03-29T04:11:05.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 -- 172.21.15.156:0/2634103498 shutdown_connections 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 --2- 172.21.15.156:0/2634103498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa94409ba50 0x7fa944160760 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 --2- 172.21.15.156:0/2634103498 >> v2:172.21.15.203:3300/0 conn(0x7fa94415cb50 0x7fa94415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 -- 172.21.15.156:0/2634103498 >> 172.21.15.156:0/2634103498 conn(0x7fa9440b3d60 msgr2=0x7fa9440b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 -- 172.21.15.156:0/2634103498 shutdown_connections 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.026+0000 7fa94943c640 1 -- 172.21.15.156:0/2634103498 wait complete. 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.027+0000 7fa94943c640 1 Processor -- start 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.027+0000 7fa94943c640 1 -- start start 2024-03-29T04:11:05.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa94943c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa94409ba50 0x7fa9440c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa94943c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 0x7fa944158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa94943c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa94415cb50 0x7fa9441f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa94943c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9440cfe40 con 0x7fa9440cd940 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa94943c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9440d0d60 con 0x7fa94415cb50 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa94943c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fa9440cffc0 con 0x7fa94409ba50 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 0x7fa944158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 0x7fa944158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36078/0 (socket says 172.21.15.156:36078) 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 -- 172.21.15.156:0/1648670727 learned_addr learned my addr 172.21.15.156:0/1648670727 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa942ffd640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa94409ba50 0x7fa9440c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9437fe640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa94415cb50 0x7fa9441f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 -- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa94409ba50 msgr2=0x7fa9440c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa94409ba50 0x7fa9440c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 -- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa94415cb50 msgr2=0x7fa9441f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa94415cb50 0x7fa9441f6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 -- 172.21.15.156:0/1648670727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9441f6bb0 con 0x7fa9440cd940 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.028+0000 7fa9427fc640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 0x7fa944158960 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa93800c3c0 tx=0x7fa93800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.029+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9380074e0 con 0x7fa9440cd940 2024-03-29T04:11:05.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.029+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa938007680 con 0x7fa9440cd940 2024-03-29T04:11:05.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.029+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938004a00 con 0x7fa9440cd940 2024-03-29T04:11:05.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.029+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9441f6e90 con 0x7fa9440cd940 2024-03-29T04:11:05.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.029+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9440c7a20 con 0x7fa9440cd940 2024-03-29T04:11:05.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.030+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9440a03c0 con 0x7fa9440cd940 2024-03-29T04:11:05.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.030+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa93801a020 con 0x7fa9440cd940 2024-03-29T04:11:05.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.031+0000 7fa92b7fe640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa93006c530 0x7fa93006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.031+0000 7fa942ffd640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa93006c530 0x7fa93006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.031+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fa93808a990 con 0x7fa9440cd940 2024-03-29T04:11:05.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.031+0000 7fa942ffd640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa93006c530 0x7fa93006e9f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa934009aa0 tx=0x7fa934009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.035+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa938058cf0 con 0x7fa9440cd940 2024-03-29T04:11:05.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.216+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fa94404f9d0 con 0x7fa93006c530 2024-03-29T04:11:05.196 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.217+0000 7fa92b7fe640 1 -- 172.21.15.156:0/1648670727 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa94404f9d0 con 0x7fa93006c530 2024-03-29T04:11:05.196 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 2 to deep-scrub 2024-03-29T04:11:05.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa93006c530 msgr2=0x7fa93006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa93006c530 0x7fa93006e9f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa934009aa0 tx=0x7fa934009190 comp rx=0 tx=0).stop 2024-03-29T04:11:05.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 msgr2=0x7fa944158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 0x7fa944158960 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa93800c3c0 tx=0x7fa93800c880 comp rx=0 tx=0).stop 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 shutdown_connections 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fa93006c530 0x7fa93006e9f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa94415cb50 0x7fa9441f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9440cd940 0x7fa944158960 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 --2- 172.21.15.156:0/1648670727 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa94409ba50 0x7fa9440c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 >> 172.21.15.156:0/1648670727 conn(0x7fa9440b3d60 msgr2=0x7fa94415bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 shutdown_connections 2024-03-29T04:11:05.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.219+0000 7fa94943c640 1 -- 172.21.15.156:0/1648670727 wait complete. 2024-03-29T04:11:05.209 INFO:tasks.ceph:Scrubbing osd.3 2024-03-29T04:11:05.209 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:05.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.335+0000 7f62f29db640 1 Processor -- start 2024-03-29T04:11:05.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.335+0000 7f62f29db640 1 -- start start 2024-03-29T04:11:05.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62f29db640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f62ec156a80 0x7f62ec071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62f29db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62f29db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec15d840 con 0x7f62ec0d0470 2024-03-29T04:11:05.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62f29db640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec06ee90 con 0x7f62ec156a80 2024-03-29T04:11:05.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62f29db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec0cfb60 con 0x7f62ec0cf5c0 2024-03-29T04:11:05.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f62ec156a80 0x7f62ec071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f62ec156a80 0x7f62ec071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36086/0 (socket says 172.21.15.156:36086) 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62f0f78640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f62ec0d0470 0x7f62ec06db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54690/0 (socket says 172.21.15.156:54690) 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 -- 172.21.15.156:0/1031535149 learned_addr learned my addr 172.21.15.156:0/1031535149 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 -- 172.21.15.156:0/1031535149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 msgr2=0x7f62ec15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62eb7fe640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 -- 172.21.15.156:0/1031535149 >> v1:172.21.15.156:6789/0 conn(0x7f62ec0d0470 legacy=0x7f62ec06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.318 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62ebfff640 1 -- 172.21.15.156:0/1031535149 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec06f0e0 con 0x7f62ec156a80 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62eaffd640 1 -- 172.21.15.156:0/1031535149 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50145340 0 0) 0x7f62e4001490 con 0x7f62ec0d0470 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.336+0000 7f62eb7fe640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.337+0000 7f62ebfff640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec071c20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009870 tx=0x7f62dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf631afd09c89c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.337+0000 7f62eaffd640 1 -- 172.21.15.156:0/1031535149 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc035020 con 0x7f62ec156a80 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.337+0000 7f62eaffd640 1 -- 172.21.15.156:0/1031535149 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc031640 con 0x7f62ec156a80 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.337+0000 7f62eaffd640 1 -- 172.21.15.156:0/1031535149 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc02d040 con 0x7f62ec156a80 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 -- 172.21.15.156:0/1031535149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 msgr2=0x7f62ec071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec071c20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009870 tx=0x7f62dc02b8f0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.319 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 -- 172.21.15.156:0/1031535149 shutdown_connections 2024-03-29T04:11:05.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 --2- 172.21.15.156:0/1031535149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec071c20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 -- 172.21.15.156:0/1031535149 >> 172.21.15.156:0/1031535149 conn(0x7f62ec0b21e0 msgr2=0x7f62ec0b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 -- 172.21.15.156:0/1031535149 shutdown_connections 2024-03-29T04:11:05.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.338+0000 7f62f29db640 1 -- 172.21.15.156:0/1031535149 wait complete. 2024-03-29T04:11:05.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.339+0000 7f62f29db640 1 Processor -- start 2024-03-29T04:11:05.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.339+0000 7f62f29db640 1 -- start start 2024-03-29T04:11:05.323 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec1f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ec0d0470 0x7f62ec09c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec09ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0d0a80 con 0x7f62ec156a80 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0d0900 con 0x7f62ec0d0470 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec0d0c00 con 0x7f62ec0cf5c0 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec09ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec09ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36102/0 (socket says 172.21.15.156:36102) 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 -- 172.21.15.156:0/2323608820 learned_addr learned my addr 172.21.15.156:0/2323608820 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62eb7fe640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ec0d0470 0x7f62ec09c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62ebfff640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec1f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 -- 172.21.15.156:0/2323608820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 msgr2=0x7f62ec1f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f62ec0cf5c0 0x7f62ec1f1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 -- 172.21.15.156:0/2323608820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ec0d0470 msgr2=0x7f62ec09c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ec0d0470 0x7f62ec09c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 -- 172.21.15.156:0/2323608820 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec1f1b40 con 0x7f62ec156a80 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f0f78640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec09ca30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c7f0 tx=0x7f62e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4018070 con 0x7f62ec156a80 2024-03-29T04:11:05.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e400a5e0 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec147470 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400aa10 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.340+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec09a5c0 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.341+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f62ec068610 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f62e401b070 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62e8ff9640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f62d806c530 0x7f62d806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f62e408a920 con 0x7f62ec156a80 2024-03-29T04:11:05.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62ebfff640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f62d806c530 0x7f62d806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62e8ff9640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f62d80723f0 0x7f62d8074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 --> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62d8074ec0 con 0x7f62d80723f0 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f62e408f2c0 con 0x7f62ec156a80 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.342+0000 7f62eb7fe640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f62d80723f0 0x7f62d8074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.343+0000 7f62ebfff640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f62d806c530 0x7f62d806e9f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f62dc036000 tx=0x7f62dc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.343+0000 7f62eb7fe640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f62d80723f0 0x7f62d8074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.343+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== osd.3 v2:172.21.15.156:6816/3904521534 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62d8074ec0 con 0x7f62d80723f0 2024-03-29T04:11:05.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.356+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 --> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f62ec073790 con 0x7f62d80723f0 2024-03-29T04:11:05.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.356+0000 7f62e8ff9640 1 -- 172.21.15.156:0/2323608820 <== osd.3 v2:172.21.15.156:6816/3904521534 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f62ec073790 con 0x7f62d80723f0 2024-03-29T04:11:05.336 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:05.336 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:05.336 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:05.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f62d80723f0 msgr2=0x7f62d8074810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6816/3904521534,v1:172.21.15.156:6817/3904521534] conn(0x7f62d80723f0 0x7f62d8074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.337 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f62d806c530 msgr2=0x7f62d806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f62d806c530 0x7f62d806e9f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f62dc036000 tx=0x7f62dc008030 comp rx=0 tx=0).stop 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 msgr2=0x7f62ec09ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 --2- 172.21.15.156:0/2323608820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62ec156a80 0x7f62ec09ca30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c7f0 tx=0x7f62e400ccb0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f0f78640 1 -- 172.21.15.156:0/2323608820 reap_dead start 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 shutdown_connections 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 >> 172.21.15.156:0/2323608820 conn(0x7f62ec0b21e0 msgr2=0x7f62ec071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 shutdown_connections 2024-03-29T04:11:05.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.357+0000 7f62f29db640 1 -- 172.21.15.156:0/2323608820 wait complete. 2024-03-29T04:11:05.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-29T04:11:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.469+0000 7f5f6eb59640 1 Processor -- start 2024-03-29T04:11:05.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.469+0000 7f5f6eb59640 1 -- start start 2024-03-29T04:11:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6eb59640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f680d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6eb59640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5f680d07c0 0x7f5f6815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6eb59640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f680b3050 con 0x7f5f680af6b0 2024-03-29T04:11:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6eb59640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f5f680b3f90 con 0x7f5f680d07c0 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6eb59640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5f680afa90 con 0x7f5f680b3b10 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5f680d07c0 0x7f5f6815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5f680d07c0 0x7f5f6815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36114/0 (socket says 172.21.15.156:36114) 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 -- 172.21.15.156:0/1723459305 learned_addr learned my addr 172.21.15.156:0/1723459305 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6c8f5640 1 --1- 172.21.15.156:0/1723459305 >> v1:172.21.15.156:6789/0 conn(0x7f5f680af6b0 0x7f5f6815c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54692/0 (socket says 172.21.15.156:54692) 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f67fff640 1 --2- 172.21.15.156:0/1723459305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f680d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 -- 172.21.15.156:0/1723459305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 msgr2=0x7f5f680d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 --2- 172.21.15.156:0/1723459305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f680d0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 -- 172.21.15.156:0/1723459305 >> v1:172.21.15.156:6789/0 conn(0x7f5f680af6b0 legacy=0x7f5f6815c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f6d0f6640 1 -- 172.21.15.156:0/1723459305 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6815d510 con 0x7f5f680d07c0 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.470+0000 7f5f677fe640 1 -- 172.21.15.156:0/1723459305 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3645903632 0 0) 0x7f5f680b3050 con 0x7f5f680af6b0 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.471+0000 7f5f6d0f6640 1 --2- 172.21.15.156:0/1723459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680d07c0 0x7f5f6815dad0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5f60009e50 tx=0x7f5f6002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64516e34ab232f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.471+0000 7f5f677fe640 1 -- 172.21.15.156:0/1723459305 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6002cb00 con 0x7f5f680d07c0 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.471+0000 7f5f677fe640 1 -- 172.21.15.156:0/1723459305 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f6002cca0 con 0x7f5f680d07c0 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.471+0000 7f5f677fe640 1 -- 172.21.15.156:0/1723459305 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f60034610 con 0x7f5f680d07c0 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 -- 172.21.15.156:0/1723459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680d07c0 msgr2=0x7f5f6815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/1723459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680d07c0 0x7f5f6815dad0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5f60009e50 tx=0x7f5f6002bf60 comp rx=0 tx=0).stop 2024-03-29T04:11:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 -- 172.21.15.156:0/1723459305 shutdown_connections 2024-03-29T04:11:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/1723459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680d07c0 0x7f5f6815dad0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/1723459305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f680d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 -- 172.21.15.156:0/1723459305 >> 172.21.15.156:0/1723459305 conn(0x7f5f680b1f30 msgr2=0x7f5f680b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 -- 172.21.15.156:0/1723459305 shutdown_connections 2024-03-29T04:11:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.472+0000 7f5f6eb59640 1 -- 172.21.15.156:0/1723459305 wait complete. 2024-03-29T04:11:05.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.473+0000 7f5f6eb59640 1 Processor -- start 2024-03-29T04:11:05.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.473+0000 7f5f6eb59640 1 -- start start 2024-03-29T04:11:05.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 0x7f5f680cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f6815e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f680d07c0 0x7f5f6815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5f680b35f0 con 0x7f5f680af6b0 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f680b3310 con 0x7f5f680d07c0 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5f680ad200 con 0x7f5f680b3b10 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 0x7f5f680cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 0x7f5f680cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36120/0 (socket says 172.21.15.156:36120) 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 -- 172.21.15.156:0/4151821195 learned_addr learned my addr 172.21.15.156:0/4151821195 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 -- 172.21.15.156:0/4151821195 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 msgr2=0x7f5f6815e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f6815e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 -- 172.21.15.156:0/4151821195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f680d07c0 msgr2=0x7f5f6815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f680d07c0 0x7f5f6815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 -- 172.21.15.156:0/4151821195 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f680cf920 con 0x7f5f680af6b0 2024-03-29T04:11:05.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6c8f5640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 0x7f5f680cddc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00c500 tx=0x7f5f5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c018070 con 0x7f5f680af6b0 2024-03-29T04:11:05.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c007580 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c004a20 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f680ce880 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.474+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f680cee40 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.475+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6809e3b0 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.476+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5f5c004bc0 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.476+0000 7f5f657fa640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5f4c06c560 0x7f5f4c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.476+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f5f5c08a590 con 0x7f5f680af6b0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.476+0000 7f5f67fff640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5f4c06c560 0x7f5f4c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.477+0000 7f5f67fff640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5f4c06c560 0x7f5f4c06ea20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5f54006fd0 tx=0x7f5f54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.480+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5f5c058f40 con 0x7f5f680af6b0 2024-03-29T04:11:05.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.656+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f5f6804f9d0 con 0x7f5f4c06c560 2024-03-29T04:11:05.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.656+0000 7f5f657fa640 1 -- 172.21.15.156:0/4151821195 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5f6804f9d0 con 0x7f5f4c06c560 2024-03-29T04:11:05.635 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 3 to deep-scrub 2024-03-29T04:11:05.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5f4c06c560 msgr2=0x7f5f4c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5f4c06c560 0x7f5f4c06ea20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5f54006fd0 tx=0x7f5f54008040 comp rx=0 tx=0).stop 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 msgr2=0x7f5f680cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 0x7f5f680cddc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00c500 tx=0x7f5f5c00c9c0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 shutdown_connections 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5f4c06c560 0x7f5f4c06ea20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f680d07c0 0x7f5f6815e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5f680b3b10 0x7f5f6815e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.638 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 --2- 172.21.15.156:0/4151821195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f680af6b0 0x7f5f680cddc0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 >> 172.21.15.156:0/4151821195 conn(0x7f5f680b1f30 msgr2=0x7f5f680b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 shutdown_connections 2024-03-29T04:11:05.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.658+0000 7f5f6eb59640 1 -- 172.21.15.156:0/4151821195 wait complete. 2024-03-29T04:11:05.649 INFO:tasks.ceph:Scrubbing osd.4 2024-03-29T04:11:05.649 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:05.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.775+0000 7fe819d52640 1 Processor -- start 2024-03-29T04:11:05.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.775+0000 7fe819d52640 1 -- start start 2024-03-29T04:11:05.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.775+0000 7fe819d52640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 0x7fe81415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.775+0000 7fe819d52640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8140b4280 con 0x7fe814070500 2024-03-29T04:11:05.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.775+0000 7fe819d52640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe81415f4a0 con 0x7fe8140b5b30 2024-03-29T04:11:05.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.775+0000 7fe819d52640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe81406e170 con 0x7fe814070120 2024-03-29T04:11:05.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe812ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fe814070500 0x7fe8140b3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54698/0 (socket says 172.21.15.156:54698) 2024-03-29T04:11:05.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe812ffd640 1 -- 172.21.15.156:0/3840949515 learned_addr learned my addr 172.21.15.156:0/3840949515 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8137fe640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 0x7fe81415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2959874198 0 0) 0x7fe8140b4280 con 0x7fe814070500 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7fc003520 con 0x7fe814070500 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1631185738 0 0) 0x7fe7fc003520 con 0x7fe814070500 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 msgr2=0x7fe81415ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 0x7fe81415ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 >> v1:172.21.15.203:6789/0 conn(0x7fe8140b5b30 legacy=0x7fe81406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81415f780 con 0x7fe814070500 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4278366210 0 0) 0x7fe808001670 con 0x7fe814070500 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 >> v1:172.21.15.156:6789/0 conn(0x7fe814070500 legacy=0x7fe8140b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7fc0039e0 0x7fe7fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc004390 0x7fe7fc004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 0x7fe7fc0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe81406e170 con 0x7fe7fc0039e0 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc00b9c0 con 0x7fe7fc004390 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe7fc00bbc0 con 0x7fe814070120 2024-03-29T04:11:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe8137fe640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7fc0039e0 0x7fe7fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe812ffd640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc004390 0x7fe7fc004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.776+0000 7fe813fff640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 0x7fe7fc0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.777+0000 7fe8137fe640 1 -- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 msgr2=0x7fe7fc0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.777+0000 7fe8137fe640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe814070120 0x7fe7fc0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.777+0000 7fe8137fe640 1 -- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc004390 msgr2=0x7fe7fc004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.777+0000 7fe8137fe640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc004390 0x7fe7fc004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.777+0000 7fe8137fe640 1 -- 172.21.15.156:0/3840949515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7fc003520 con 0x7fe7fc0039e0 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.777+0000 7fe8137fe640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7fc0039e0 0x7fe7fc003dc0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe804009ac0 tx=0x7fe7fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4bd4d879d703e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804039070 con 0x7fe7fc0039e0 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe80402c6b0 con 0x7fe7fc0039e0 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe8127fc640 1 -- 172.21.15.156:0/3840949515 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804009690 con 0x7fe7fc0039e0 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe819d52640 1 -- 172.21.15.156:0/3840949515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7fc0039e0 msgr2=0x7fe7fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe819d52640 1 --2- 172.21.15.156:0/3840949515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7fc0039e0 0x7fe7fc003dc0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe804009ac0 tx=0x7fe7fc005830 comp rx=0 tx=0).stop 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe813fff640 1 -- 172.21.15.156:0/3840949515 reap_dead start 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe819d52640 1 -- 172.21.15.156:0/3840949515 shutdown_connections 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe819d52640 1 -- 172.21.15.156:0/3840949515 >> 172.21.15.156:0/3840949515 conn(0x7fe814072160 msgr2=0x7fe814072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe819d52640 1 -- 172.21.15.156:0/3840949515 shutdown_connections 2024-03-29T04:11:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.778+0000 7fe819d52640 1 -- 172.21.15.156:0/3840949515 wait complete. 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.779+0000 7fe819d52640 1 Processor -- start 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.779+0000 7fe819d52640 1 -- start start 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe819d52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0039e0 0x7fe8140ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe819d52640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe814159cd0 0x7fe8140cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe819d52640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8140d0460 0x7fe81415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe819d52640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe81406ccd0 con 0x7fe814159cd0 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe819d52640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe81406e3c0 con 0x7fe7fc0039e0 2024-03-29T04:11:05.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe819d52640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fe81406ce50 con 0x7fe8140d0460 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe812ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe814159cd0 0x7fe8140cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe812ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe814159cd0 0x7fe8140cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36136/0 (socket says 172.21.15.156:36136) 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe812ffd640 1 -- 172.21.15.156:0/32410423 learned_addr learned my addr 172.21.15.156:0/32410423 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe813fff640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8140d0460 0x7fe81415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.780+0000 7fe8137fe640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0039e0 0x7fe8140ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe812ffd640 1 -- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8140d0460 msgr2=0x7fe81415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe812ffd640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8140d0460 0x7fe81415b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe812ffd640 1 -- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0039e0 msgr2=0x7fe8140ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe812ffd640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0039e0 0x7fe8140ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe812ffd640 1 -- 172.21.15.156:0/32410423 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8141478f0 con 0x7fe814159cd0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe8137fe640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0039e0 0x7fe8140ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe812ffd640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe814159cd0 0x7fe8140cff20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe808007340 tx=0x7fe808007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80800f860 con 0x7fe814159cd0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe808018070 con 0x7fe814159cd0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8140cee80 con 0x7fe814159cd0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80800fc60 con 0x7fe814159cd0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.781+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8140cf440 con 0x7fe814159cd0 2024-03-29T04:11:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.782+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe81415e6d0 con 0x7fe814159cd0 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.782+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe80801d460 con 0x7fe814159cd0 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe7fbfff640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe80006c560 0x7fe80006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fe80808b290 con 0x7fe814159cd0 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe8137fe640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe80006c560 0x7fe80006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe7fbfff640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fe800072420 0x7fe800074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 --> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe80800f020 con 0x7fe800072420 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe808092050 con 0x7fe814159cd0 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe813fff640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fe800072420 0x7fe800074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe8137fe640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe80006c560 0x7fe80006ea20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe804009ac0 tx=0x7fe804030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.783+0000 7fe813fff640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fe800072420 0x7fe800074840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.784+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== osd.4 v2:172.21.15.203:6816/3017904752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe80800f020 con 0x7fe800072420 2024-03-29T04:11:05.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.797+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 --> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe814068610 con 0x7fe800072420 2024-03-29T04:11:05.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe7fbfff640 1 -- 172.21.15.156:0/32410423 <== osd.4 v2:172.21.15.203:6816/3017904752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe814068610 con 0x7fe800072420 2024-03-29T04:11:05.777 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:05.777 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:05.777 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:05.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fe800072420 msgr2=0x7fe800074840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.203:6816/3017904752,v1:172.21.15.203:6818/3017904752] conn(0x7fe800072420 0x7fe800074840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe80006c560 msgr2=0x7fe80006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fe80006c560 0x7fe80006ea20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe804009ac0 tx=0x7fe804030000 comp rx=0 tx=0).stop 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe814159cd0 msgr2=0x7fe8140cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 --2- 172.21.15.156:0/32410423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe814159cd0 0x7fe8140cff20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe808007340 tx=0x7fe808007800 comp rx=0 tx=0).stop 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe813fff640 1 -- 172.21.15.156:0/32410423 reap_dead start 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 shutdown_connections 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 >> 172.21.15.156:0/32410423 conn(0x7fe814072160 msgr2=0x7fe8140b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 shutdown_connections 2024-03-29T04:11:05.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.798+0000 7fe819d52640 1 -- 172.21.15.156:0/32410423 wait complete. 2024-03-29T04:11:05.788 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-29T04:11:05.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.909+0000 7fed2e875640 1 Processor -- start 2024-03-29T04:11:05.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2e875640 1 -- start start 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2e875640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed2806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2e875640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fed2806cb10 0x7fed280cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2e875640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed280d2df0 con 0x7fed2815e920 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2e875640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fed280cf7a0 con 0x7fed2806cb10 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2e875640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fed280cf190 con 0x7fed280d2120 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed2ce12640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fed2806cb10 0x7fed280cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed27fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fed2815e920 0x7fed280d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54702/0 (socket says 172.21.15.156:54702) 2024-03-29T04:11:05.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.910+0000 7fed27fff640 1 -- 172.21.15.156:0/3149662108 learned_addr learned my addr 172.21.15.156:0/3149662108 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.911+0000 7fed277fe640 1 --2- 172.21.15.156:0/3149662108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed2806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.911+0000 7fed2ce12640 1 -- 172.21.15.156:0/3149662108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 msgr2=0x7fed2806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.911+0000 7fed2ce12640 1 --2- 172.21.15.156:0/3149662108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed2806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.911+0000 7fed2ce12640 1 -- 172.21.15.156:0/3149662108 >> v1:172.21.15.156:6789/0 conn(0x7fed2815e920 legacy=0x7fed280d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.911+0000 7fed2ce12640 1 -- 172.21.15.156:0/3149662108 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed280d3080 con 0x7fed2806cb10 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2ce12640 1 --2- 172.21.15.156:0/3149662108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed280cfc30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fed20009d90 tx=0x7fed2002bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a565d225e9faa62f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed26ffd640 1 -- 172.21.15.156:0/3149662108 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20039070 con 0x7fed2806cb10 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed26ffd640 1 -- 172.21.15.156:0/3149662108 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed2002ca10 con 0x7fed2806cb10 2024-03-29T04:11:05.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed26ffd640 1 -- 172.21.15.156:0/3149662108 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2002cde0 con 0x7fed2806cb10 2024-03-29T04:11:05.895 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 -- 172.21.15.156:0/3149662108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 msgr2=0x7fed280cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 --2- 172.21.15.156:0/3149662108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed280cfc30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fed20009d90 tx=0x7fed2002bce0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 -- 172.21.15.156:0/3149662108 shutdown_connections 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 --2- 172.21.15.156:0/3149662108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed280cfc30 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 --2- 172.21.15.156:0/3149662108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed2806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 -- 172.21.15.156:0/3149662108 >> 172.21.15.156:0/3149662108 conn(0x7fed2809d7a0 msgr2=0x7fed280d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 -- 172.21.15.156:0/3149662108 shutdown_connections 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.912+0000 7fed2e875640 1 -- 172.21.15.156:0/3149662108 wait complete. 2024-03-29T04:11:05.896 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.913+0000 7fed2e875640 1 Processor -- start 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 -- start start 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed281578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed280b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed2815e920 0x7fed280cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fed280d34e0 con 0x7fed2806cb10 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed280d3360 con 0x7fed2815e920 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed2e875640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fed280d0170 con 0x7fed280d2120 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed27fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed281578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed27fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed281578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36144/0 (socket says 172.21.15.156:36144) 2024-03-29T04:11:05.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.914+0000 7fed27fff640 1 -- 172.21.15.156:0/3551962825 learned_addr learned my addr 172.21.15.156:0/3551962825 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed2ce12640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed2815e920 0x7fed280cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed277fe640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed280b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed27fff640 1 -- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 msgr2=0x7fed280b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed27fff640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed280b97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed27fff640 1 -- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed2815e920 msgr2=0x7fed280cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed27fff640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed2815e920 0x7fed280cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed27fff640 1 -- 172.21.15.156:0/3551962825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed28157df0 con 0x7fed2806cb10 2024-03-29T04:11:05.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed2ce12640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed2815e920 0x7fed280cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed27fff640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed281578b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c780 tx=0x7fed1800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18018070 con 0x7fed2806cb10 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed180072d0 con 0x7fed2806cb10 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed280c9820 con 0x7fed2806cb10 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180047e0 con 0x7fed2806cb10 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.915+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed280c9de0 con 0x7fed2806cb10 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.916+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed280a2480 con 0x7fed2806cb10 2024-03-29T04:11:05.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.917+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fed1801b070 con 0x7fed2806cb10 2024-03-29T04:11:05.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.917+0000 7fed24ff9640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fed1406c530 0x7fed1406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:05.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.917+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fed1808a740 con 0x7fed2806cb10 2024-03-29T04:11:05.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.917+0000 7fed277fe640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fed1406c530 0x7fed1406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:05.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.917+0000 7fed277fe640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fed1406c530 0x7fed1406e9f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed280b9700 tx=0x7fed1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:05.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:05.921+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fed180591b0 con 0x7fed2806cb10 2024-03-29T04:11:06.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.100+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fed2804f9d0 con 0x7fed1406c530 2024-03-29T04:11:06.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.100+0000 7fed24ff9640 1 -- 172.21.15.156:0/3551962825 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fed2804f9d0 con 0x7fed1406c530 2024-03-29T04:11:06.080 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 4 to deep-scrub 2024-03-29T04:11:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.102+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fed1406c530 msgr2=0x7fed1406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.102+0000 7fed2e875640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fed1406c530 0x7fed1406e9f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed280b9700 tx=0x7fed1c008040 comp rx=0 tx=0).stop 2024-03-29T04:11:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.102+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 msgr2=0x7fed281578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.102+0000 7fed2e875640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed281578b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c780 tx=0x7fed1800cc40 comp rx=0 tx=0).stop 2024-03-29T04:11:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 shutdown_connections 2024-03-29T04:11:06.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fed1406c530 0x7fed1406e9f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed2815e920 0x7fed280cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fed280d2120 0x7fed280b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 --2- 172.21.15.156:0/3551962825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed2806cb10 0x7fed281578b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 >> 172.21.15.156:0/3551962825 conn(0x7fed2809d7a0 msgr2=0x7fed2815ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 shutdown_connections 2024-03-29T04:11:06.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.103+0000 7fed2e875640 1 -- 172.21.15.156:0/3551962825 wait complete. 2024-03-29T04:11:06.093 INFO:tasks.ceph:Scrubbing osd.5 2024-03-29T04:11:06.093 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:06.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.218+0000 7fbfd4966640 1 Processor -- start 2024-03-29T04:11:06.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.218+0000 7fbfd4966640 1 -- start start 2024-03-29T04:11:06.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.219+0000 7fbfd4966640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbfd0070180 0x7fbfd006e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.219+0000 7fbfd4966640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.219+0000 7fbfd4966640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd015eed0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.219+0000 7fbfd4966640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd015f1b0 con 0x7fbfd0070180 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.219+0000 7fbfd4966640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd015f490 con 0x7fbfd0070560 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfce59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbfd0070180 0x7fbfd006e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfced9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbfd00b5d80 0x7fbfd006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54706/0 (socket says 172.21.15.156:54706) 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfced9d640 1 -- 172.21.15.156:0/3575797548 learned_addr learned my addr 172.21.15.156:0/3575797548 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfcdd9b640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfce59c640 1 -- 172.21.15.156:0/3575797548 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 msgr2=0x7fbfd00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfce59c640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd00b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.200 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfce59c640 1 -- 172.21.15.156:0/3575797548 >> v1:172.21.15.156:6789/0 conn(0x7fbfd00b5d80 legacy=0x7fbfd006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfce59c640 1 -- 172.21.15.156:0/3575797548 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd015f770 con 0x7fbfd0070180 2024-03-29T04:11:06.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfcd59a640 1 -- 172.21.15.156:0/3575797548 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024352375 0 0) 0x7fbfd015eed0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.220+0000 7fbfcdd9b640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfce59c640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd0070180 0x7fbfd006e170 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009d90 tx=0x7fbfc002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa64048c6f553af server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfcd59a640 1 -- 172.21.15.156:0/3575797548 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0039070 con 0x7fbfd0070180 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfcd59a640 1 -- 172.21.15.156:0/3575797548 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc002cbc0 con 0x7fbfd0070180 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfcd59a640 1 -- 172.21.15.156:0/3575797548 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0034440 con 0x7fbfd0070180 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 -- 172.21.15.156:0/3575797548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd0070180 msgr2=0x7fbfd006e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd0070180 0x7fbfd006e170 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009d90 tx=0x7fbfc002be10 comp rx=0 tx=0).stop 2024-03-29T04:11:06.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 -- 172.21.15.156:0/3575797548 shutdown_connections 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 --2- 172.21.15.156:0/3575797548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd0070180 0x7fbfd006e170 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 -- 172.21.15.156:0/3575797548 >> 172.21.15.156:0/3575797548 conn(0x7fbfd0072160 msgr2=0x7fbfd0072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 -- 172.21.15.156:0/3575797548 shutdown_connections 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.221+0000 7fbfd4966640 1 -- 172.21.15.156:0/3575797548 wait complete. 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.223+0000 7fbfd4966640 1 Processor -- start 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.223+0000 7fbfd4966640 1 -- start start 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.223+0000 7fbfd4966640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfd0070180 0x7fbfd015d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.223+0000 7fbfd4966640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd01f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.223+0000 7fbfd4966640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd00b5d80 0x7fbfd015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfd4966640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd006ccd0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfd4966640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd015fa50 con 0x7fbfd0070180 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfd4966640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd006ce50 con 0x7fbfd0070560 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd00b5d80 0x7fbfd015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd00b5d80 0x7fbfd015a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36162/0 (socket says 172.21.15.156:36162) 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 -- 172.21.15.156:0/2252906301 learned_addr learned my addr 172.21.15.156:0/2252906301 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfce59c640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfd0070180 0x7fbfd015d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfcdd9b640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd01f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 -- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 msgr2=0x7fbfd01f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbfd0070560 0x7fbfd01f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 -- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfd0070180 msgr2=0x7fbfd015d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfd0070180 0x7fbfd015d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 -- 172.21.15.156:0/2252906301 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd01f4ff0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfced9d640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd00b5d80 0x7fbfd015a6d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800c450 tx=0x7fbfc800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80074e0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc80047e0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd015d7f0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8004bd0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.224+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd015ddb0 con 0x7fbfd00b5d80 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.225+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfd0068610 con 0x7fbfd00b5d80 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbfc8011070 con 0x7fbfd00b5d80 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfbeffd640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbfb006c530 0x7fbfb006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fbfc808ae80 con 0x7fbfd00b5d80 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfce59c640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbfb006c530 0x7fbfb006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfbeffd640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fbfb00723f0 0x7fbfb0074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 --> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfb0074ec0 con 0x7fbfb00723f0 2024-03-29T04:11:06.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbfc808d580 con 0x7fbfd00b5d80 2024-03-29T04:11:06.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.226+0000 7fbfcdd9b640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fbfb00723f0 0x7fbfb0074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.227+0000 7fbfce59c640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbfb006c530 0x7fbfb006e9f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0036000 tx=0x7fbfc0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.227+0000 7fbfcdd9b640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fbfb00723f0 0x7fbfb0074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.228+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== osd.5 v2:172.21.15.203:6808/2129623277 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbfb0074ec0 con 0x7fbfb00723f0 2024-03-29T04:11:06.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 --> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbfd00b25e0 con 0x7fbfb00723f0 2024-03-29T04:11:06.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfbeffd640 1 -- 172.21.15.156:0/2252906301 <== osd.5 v2:172.21.15.203:6808/2129623277 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbfd00b25e0 con 0x7fbfb00723f0 2024-03-29T04:11:06.220 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:06.220 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:06.221 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fbfb00723f0 msgr2=0x7fbfb0074810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.203:6808/2129623277,v1:172.21.15.203:6809/2129623277] conn(0x7fbfb00723f0 0x7fbfb0074810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbfb006c530 msgr2=0x7fbfb006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7fbfb006c530 0x7fbfb006e9f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0036000 tx=0x7fbfc0030000 comp rx=0 tx=0).stop 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd00b5d80 msgr2=0x7fbfd015a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 --2- 172.21.15.156:0/2252906301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfd00b5d80 0x7fbfd015a6d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800c450 tx=0x7fbfc800c910 comp rx=0 tx=0).stop 2024-03-29T04:11:06.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfced9d640 1 -- 172.21.15.156:0/2252906301 reap_dead start 2024-03-29T04:11:06.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 shutdown_connections 2024-03-29T04:11:06.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 >> 172.21.15.156:0/2252906301 conn(0x7fbfd0072160 msgr2=0x7fbfd01584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.241+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 shutdown_connections 2024-03-29T04:11:06.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.242+0000 7fbfd4966640 1 -- 172.21.15.156:0/2252906301 wait complete. 2024-03-29T04:11:06.231 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-29T04:11:06.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.353+0000 7f7b3b0d9640 1 Processor -- start 2024-03-29T04:11:06.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.353+0000 7f7b3b0d9640 1 -- start start 2024-03-29T04:11:06.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b3b0d9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7b340af6b0 0x7f7b340b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b3b0d9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7b340d0990 0x7f7b3415ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b3b0d9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b340b3050 con 0x7f7b3415c990 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b3b0d9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7b340afa90 con 0x7f7b340d0990 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b3b0d9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7b3415d510 con 0x7f7b340af6b0 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7b340d0990 0x7f7b3415ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7b340d0990 0x7f7b3415ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36176/0 (socket says 172.21.15.156:36176) 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 -- 172.21.15.156:0/3635476052 learned_addr learned my addr 172.21.15.156:0/3635476052 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b33fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7b3415c990 0x7f7b340d0280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54710/0 (socket says 172.21.15.156:54710) 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b38e75640 1 --2- 172.21.15.156:0/3635476052 >> v2:172.21.15.203:3300/0 conn(0x7f7b340af6b0 0x7f7b340b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.335 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 -- 172.21.15.156:0/3635476052 >> v2:172.21.15.203:3300/0 conn(0x7f7b340af6b0 msgr2=0x7f7b340b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 --2- 172.21.15.156:0/3635476052 >> v2:172.21.15.203:3300/0 conn(0x7f7b340af6b0 0x7f7b340b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 -- 172.21.15.156:0/3635476052 >> v1:172.21.15.156:6789/0 conn(0x7f7b3415c990 legacy=0x7f7b340d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.354+0000 7f7b39676640 1 -- 172.21.15.156:0/3635476052 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3415d7f0 con 0x7f7b340d0990 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.355+0000 7f7b39676640 1 --2- 172.21.15.156:0/3635476052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b3415ce40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7b34073a00 tx=0x7f7b2c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6231e283e988b4a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.355+0000 7f7b337fe640 1 -- 172.21.15.156:0/3635476052 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c02cb20 con 0x7f7b340d0990 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.355+0000 7f7b337fe640 1 -- 172.21.15.156:0/3635476052 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b2c02ccc0 con 0x7f7b340d0990 2024-03-29T04:11:06.336 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.355+0000 7f7b337fe640 1 -- 172.21.15.156:0/3635476052 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c034610 con 0x7f7b340d0990 2024-03-29T04:11:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/3635476052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 msgr2=0x7f7b3415ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/3635476052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b3415ce40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7b34073a00 tx=0x7f7b2c02bf80 comp rx=0 tx=0).stop 2024-03-29T04:11:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/3635476052 shutdown_connections 2024-03-29T04:11:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/3635476052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b3415ce40 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/3635476052 >> v2:172.21.15.203:3300/0 conn(0x7f7b340af6b0 0x7f7b340b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.339 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/3635476052 >> 172.21.15.156:0/3635476052 conn(0x7f7b340b1f30 msgr2=0x7f7b340b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/3635476052 shutdown_connections 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.356+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/3635476052 wait complete. 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.357+0000 7f7b3b0d9640 1 Processor -- start 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.357+0000 7f7b3b0d9640 1 -- start start 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b3b0d9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b340af6b0 0x7f7b3409dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b3b0d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b340cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b3b0d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3415c990 0x7f7b34155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b3b0d9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b340ad200 con 0x7f7b340d0990 2024-03-29T04:11:06.340 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b3b0d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b340b35f0 con 0x7f7b3415c990 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b3b0d9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3415dad0 con 0x7f7b340af6b0 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b340cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b340cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36184/0 (socket says 172.21.15.156:36184) 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 -- 172.21.15.156:0/2249991957 learned_addr learned my addr 172.21.15.156:0/2249991957 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b39676640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3415c990 0x7f7b34155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b38e75640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b340af6b0 0x7f7b3409dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 -- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b340af6b0 msgr2=0x7f7b3409dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.341 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b340af6b0 0x7f7b3409dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 -- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3415c990 msgr2=0x7f7b34155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3415c990 0x7f7b34155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.358+0000 7f7b33fff640 1 -- 172.21.15.156:0/2249991957 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3409cf80 con 0x7f7b340d0990 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.359+0000 7f7b33fff640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b340cf860 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000c3c0 tx=0x7f7b2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.359+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b200074e0 con 0x7f7b340d0990 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.359+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b20007680 con 0x7f7b340d0990 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.359+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20004a00 con 0x7f7b340d0990 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.359+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3409d1f0 con 0x7f7b340d0990 2024-03-29T04:11:06.342 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.359+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b341f3320 con 0x7f7b340d0990 2024-03-29T04:11:06.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.360+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3409e3b0 con 0x7f7b340d0990 2024-03-29T04:11:06.343 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.360+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7b2001a020 con 0x7f7b340d0990 2024-03-29T04:11:06.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.361+0000 7f7b317fa640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b1806c530 0x7f7b1806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.361+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f7b2008aae0 con 0x7f7b340d0990 2024-03-29T04:11:06.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.361+0000 7f7b38e75640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b1806c530 0x7f7b1806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.361+0000 7f7b38e75640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b1806c530 0x7f7b1806e9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7b28006fd0 tx=0x7f7b28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.344 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.365+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7b20059490 con 0x7f7b340d0990 2024-03-29T04:11:06.520 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.541+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f7b3404f9d0 con 0x7f7b1806c530 2024-03-29T04:11:06.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.541+0000 7f7b317fa640 1 -- 172.21.15.156:0/2249991957 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7b3404f9d0 con 0x7f7b1806c530 2024-03-29T04:11:06.521 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 5 to deep-scrub 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b1806c530 msgr2=0x7f7b1806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b1806c530 0x7f7b1806e9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7b28006fd0 tx=0x7f7b28008040 comp rx=0 tx=0).stop 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 msgr2=0x7f7b340cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b340cf860 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000c3c0 tx=0x7f7b2000c880 comp rx=0 tx=0).stop 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 shutdown_connections 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f7b1806c530 0x7f7b1806e9f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3415c990 0x7f7b34155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b340d0990 0x7f7b340cf860 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 --2- 172.21.15.156:0/2249991957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b340af6b0 0x7f7b3409dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 >> 172.21.15.156:0/2249991957 conn(0x7f7b340b1f30 msgr2=0x7f7b3406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 shutdown_connections 2024-03-29T04:11:06.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.543+0000 7f7b3b0d9640 1 -- 172.21.15.156:0/2249991957 wait complete. 2024-03-29T04:11:06.534 INFO:tasks.ceph:Scrubbing osd.6 2024-03-29T04:11:06.534 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:06.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.660+0000 7f5aa7950640 1 Processor -- start 2024-03-29T04:11:06.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.660+0000 7f5aa7950640 1 -- start start 2024-03-29T04:11:06.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa7950640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa015b460 0x7f5aa00ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa7950640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 0x7f5aa0071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa7950640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa00729b0 con 0x7f5aa0071810 2024-03-29T04:11:06.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa7950640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa0072c90 con 0x7f5aa00d0b80 2024-03-29T04:11:06.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa7950640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa0072f70 con 0x7f5aa015b460 2024-03-29T04:11:06.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa56ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa015b460 0x7f5aa00ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa5eed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 0x7f5aa0071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa5eed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 0x7f5aa0071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:45094/0 (socket says 172.21.15.156:45094) 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.661+0000 7f5aa5eed640 1 -- 172.21.15.156:0/3509976708 learned_addr learned my addr 172.21.15.156:0/3509976708 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.662+0000 7f5aa56ec640 1 -- 172.21.15.156:0/3509976708 >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 msgr2=0x7f5aa0071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.662+0000 7f5aa56ec640 1 --2- 172.21.15.156:0/3509976708 >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 0x7f5aa0071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.662+0000 7f5aa56ec640 1 -- 172.21.15.156:0/3509976708 >> v1:172.21.15.203:6789/0 conn(0x7f5aa0071810 legacy=0x7f5aa00d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.662+0000 7f5aa56ec640 1 -- 172.21.15.156:0/3509976708 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa0072680 con 0x7f5aa015b460 2024-03-29T04:11:06.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.662+0000 7f5aa5eed640 1 --2- 172.21.15.156:0/3509976708 >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 0x7f5aa0071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:11:06.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa56ec640 1 --2- 172.21.15.156:0/3509976708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa015b460 0x7f5aa00ada60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009d90 tx=0x7f5a9c02bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cd3517577c50baf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5a97fff640 1 -- 172.21.15.156:0/3509976708 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c039070 con 0x7f5aa015b460 2024-03-29T04:11:06.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5a97fff640 1 -- 172.21.15.156:0/3509976708 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9c02cb30 con 0x7f5aa015b460 2024-03-29T04:11:06.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5a97fff640 1 -- 172.21.15.156:0/3509976708 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c034440 con 0x7f5aa015b460 2024-03-29T04:11:06.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 -- 172.21.15.156:0/3509976708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa015b460 msgr2=0x7f5aa00ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 --2- 172.21.15.156:0/3509976708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa015b460 0x7f5aa00ada60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009d90 tx=0x7f5a9c02bd80 comp rx=0 tx=0).stop 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 -- 172.21.15.156:0/3509976708 shutdown_connections 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 --2- 172.21.15.156:0/3509976708 >> v2:172.21.15.203:3300/0 conn(0x7f5aa00d0b80 0x7f5aa0071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 --2- 172.21.15.156:0/3509976708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa015b460 0x7f5aa00ada60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 -- 172.21.15.156:0/3509976708 >> 172.21.15.156:0/3509976708 conn(0x7f5aa00b21e0 msgr2=0x7f5aa00b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 -- 172.21.15.156:0/3509976708 shutdown_connections 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.663+0000 7f5aa7950640 1 -- 172.21.15.156:0/3509976708 wait complete. 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.664+0000 7f5aa7950640 1 Processor -- start 2024-03-29T04:11:06.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 -- start start 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa0071810 0x7f5aa015f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa00d0b80 0x7f5aa014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa015b460 0x7f5aa0145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa00b37c0 con 0x7f5aa015b460 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa00b3640 con 0x7f5aa0071810 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa7950640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa00b3940 con 0x7f5aa00d0b80 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa5eed640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa015b460 0x7f5aa0145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.649 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa5eed640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa015b460 0x7f5aa0145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36196/0 (socket says 172.21.15.156:36196) 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa5eed640 1 -- 172.21.15.156:0/2171797952 learned_addr learned my addr 172.21.15.156:0/2171797952 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa4eeb640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa00d0b80 0x7f5aa014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.665+0000 7f5aa56ec640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa0071810 0x7f5aa015f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa5eed640 1 -- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa00d0b80 msgr2=0x7f5aa014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa5eed640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5aa00d0b80 0x7f5aa014e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa5eed640 1 -- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa0071810 msgr2=0x7f5aa015f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa5eed640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aa0071810 0x7f5aa015f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa5eed640 1 -- 172.21.15.156:0/2171797952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa015e2e0 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa5eed640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa015b460 0x7f5aa0145390 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e8d0 tx=0x7f5a9800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98017070 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a98004020 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa015e5c0 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a980043f0 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.666+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa015eb80 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.667+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5aa0068610 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5a9800c940 con 0x7f5aa015b460 2024-03-29T04:11:06.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5a95ffb640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8406c560 0x7f5a8406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f5a9808b500 con 0x7f5aa015b460 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5aa56ec640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8406c560 0x7f5a8406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5a95ffb640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f5a84072420 0x7f5a84074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 --> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a98016ec0 con 0x7f5a84072420 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5a98092050 con 0x7f5aa015b460 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.668+0000 7f5aa4eeb640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f5a84072420 0x7f5a84074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.669+0000 7f5aa56ec640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8406c560 0x7f5a8406ea20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c036000 tx=0x7f5a9c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.669+0000 7f5aa4eeb640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f5a84072420 0x7f5a84074840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.670+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== osd.6 v2:172.21.15.203:6800/505413700 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a98016ec0 con 0x7f5a84072420 2024-03-29T04:11:06.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.683+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 --> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5aa006f3f0 con 0x7f5a84072420 2024-03-29T04:11:06.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.683+0000 7f5a95ffb640 1 -- 172.21.15.156:0/2171797952 <== osd.6 v2:172.21.15.203:6800/505413700 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5aa006f3f0 con 0x7f5a84072420 2024-03-29T04:11:06.663 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:06.663 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:06.663 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:06.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f5a84072420 msgr2=0x7f5a84074840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.203:6800/505413700,v1:172.21.15.203:6801/505413700] conn(0x7f5a84072420 0x7f5a84074840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8406c560 msgr2=0x7f5a8406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f5a8406c560 0x7f5a8406ea20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c036000 tx=0x7f5a9c030000 comp rx=0 tx=0).stop 2024-03-29T04:11:06.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa015b460 msgr2=0x7f5aa0145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 --2- 172.21.15.156:0/2171797952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5aa015b460 0x7f5aa0145390 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e8d0 tx=0x7f5a9800ed90 comp rx=0 tx=0).stop 2024-03-29T04:11:06.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa5eed640 1 -- 172.21.15.156:0/2171797952 reap_dead start 2024-03-29T04:11:06.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 shutdown_connections 2024-03-29T04:11:06.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 >> 172.21.15.156:0/2171797952 conn(0x7f5aa00b21e0 msgr2=0x7f5aa00ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 shutdown_connections 2024-03-29T04:11:06.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.684+0000 7f5aa7950640 1 -- 172.21.15.156:0/2171797952 wait complete. 2024-03-29T04:11:06.673 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-29T04:11:06.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.795+0000 7f68fa8c5640 1 Processor -- start 2024-03-29T04:11:06.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.795+0000 7f68fa8c5640 1 -- start start 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68fa8c5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f68f415e920 0x7f68f40d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68fa8c5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f68f40d2120 0x7f68f406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68fa8c5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f40cf190 con 0x7f68f406cb10 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68fa8c5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f68f40cf7a0 con 0x7f68f415e920 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68fa8c5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f68f40d3080 con 0x7f68f40d2120 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68f3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f68f415e920 0x7f68f40d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68f3fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f68f415e920 0x7f68f40d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36198/0 (socket says 172.21.15.156:36198) 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.796+0000 7f68f3fff640 1 -- 172.21.15.156:0/1155416454 learned_addr learned my addr 172.21.15.156:0/1155416454 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f37fe640 1 --2- 172.21.15.156:0/1155416454 >> v2:172.21.15.203:3300/0 conn(0x7f68f40d2120 0x7f68f406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f3fff640 1 -- 172.21.15.156:0/1155416454 >> v2:172.21.15.203:3300/0 conn(0x7f68f40d2120 msgr2=0x7f68f406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f3fff640 1 --2- 172.21.15.156:0/1155416454 >> v2:172.21.15.203:3300/0 conn(0x7f68f40d2120 0x7f68f406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f3fff640 1 -- 172.21.15.156:0/1155416454 >> v1:172.21.15.203:6789/0 conn(0x7f68f406cb10 legacy=0x7f68f40cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f3fff640 1 -- 172.21.15.156:0/1155416454 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f40cf430 con 0x7f68f415e920 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f37fe640 1 --2- 172.21.15.156:0/1155416454 >> v2:172.21.15.203:3300/0 conn(0x7f68f40d2120 0x7f68f406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:06.778 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.797+0000 7f68f3fff640 1 --2- 172.21.15.156:0/1155416454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f415e920 0x7f68f40d26e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f68f408c830 tx=0x7f68e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7141f025faf49967 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68f2ffd640 1 -- 172.21.15.156:0/1155416454 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4035020 con 0x7f68f415e920 2024-03-29T04:11:06.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68f2ffd640 1 -- 172.21.15.156:0/1155416454 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e4031870 con 0x7f68f415e920 2024-03-29T04:11:06.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68f2ffd640 1 -- 172.21.15.156:0/1155416454 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4031c40 con 0x7f68f415e920 2024-03-29T04:11:06.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1155416454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f415e920 msgr2=0x7f68f40d26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1155416454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f415e920 0x7f68f40d26e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f68f408c830 tx=0x7f68e402b910 comp rx=0 tx=0).stop 2024-03-29T04:11:06.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1155416454 shutdown_connections 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1155416454 >> v2:172.21.15.203:3300/0 conn(0x7f68f40d2120 0x7f68f406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1155416454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f415e920 0x7f68f40d26e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1155416454 >> 172.21.15.156:0/1155416454 conn(0x7f68f409d7a0 msgr2=0x7f68f40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1155416454 shutdown_connections 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.798+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1155416454 wait complete. 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.799+0000 7f68fa8c5640 1 Processor -- start 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 -- start start 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68f406cb10 0x7f68f409f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 0x7f68f40cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68f415e920 0x7f68f414ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40d04c0 con 0x7f68f40d2120 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40d0340 con 0x7f68f415e920 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68fa8c5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f68f40d2c20 con 0x7f68f406cb10 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 0x7f68f40cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 0x7f68f40cbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36204/0 (socket says 172.21.15.156:36204) 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.800+0000 7f68f37fe640 1 -- 172.21.15.156:0/1417409987 learned_addr learned my addr 172.21.15.156:0/1417409987 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f3fff640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68f406cb10 0x7f68f409f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f8e62640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68f415e920 0x7f68f414ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f37fe640 1 -- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68f406cb10 msgr2=0x7f68f409f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f37fe640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68f406cb10 0x7f68f409f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f37fe640 1 -- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68f415e920 msgr2=0x7f68f414ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f37fe640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68f415e920 0x7f68f414ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f37fe640 1 -- 172.21.15.156:0/1417409987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f40cc940 con 0x7f68f40d2120 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f3fff640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68f406cb10 0x7f68f409f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f37fe640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 0x7f68f40cbf30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f68e800c7f0 tx=0x7f68e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e8015070 con 0x7f68f40d2120 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e8007a30 con 0x7f68f40d2120 2024-03-29T04:11:06.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e8007e00 con 0x7f68f40d2120 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f41e8130 con 0x7f68f40d2120 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.801+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f4150b10 con 0x7f68f40d2120 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.802+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f4068fc0 con 0x7f68f40d2120 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.802+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f68e8018070 con 0x7f68f40d2120 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.803+0000 7f68f0ff9640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f68e006c530 0x7f68e006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.803+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f68e808b3b0 con 0x7f68f40d2120 2024-03-29T04:11:06.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.803+0000 7f68f3fff640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f68e006c530 0x7f68e006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:06.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.803+0000 7f68f3fff640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f68e006c530 0x7f68e006e9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f68e4036000 tx=0x7f68e4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:06.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.807+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f68f4068fc0 con 0x7f68f40d2120 2024-03-29T04:11:06.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.980+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f68f406c8d0 con 0x7f68e006c530 2024-03-29T04:11:06.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.981+0000 7f68f0ff9640 1 -- 172.21.15.156:0/1417409987 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f68f406c8d0 con 0x7f68e006c530 2024-03-29T04:11:06.960 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 6 to deep-scrub 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f68e006c530 msgr2=0x7f68e006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f68e006c530 0x7f68e006e9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f68e4036000 tx=0x7f68e4008030 comp rx=0 tx=0).stop 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 msgr2=0x7f68f40cbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 0x7f68f40cbf30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f68e800c7f0 tx=0x7f68e800cbf0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 shutdown_connections 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f68e006c530 0x7f68e006e9f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68f415e920 0x7f68f414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.962 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68f40d2120 0x7f68f40cbf30 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 --2- 172.21.15.156:0/1417409987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68f406cb10 0x7f68f409f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:06.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 >> 172.21.15.156:0/1417409987 conn(0x7f68f409d7a0 msgr2=0x7f68f40b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:06.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 shutdown_connections 2024-03-29T04:11:06.963 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:06.982+0000 7f68fa8c5640 1 -- 172.21.15.156:0/1417409987 wait complete. 2024-03-29T04:11:06.972 INFO:tasks.ceph:Scrubbing osd.7 2024-03-29T04:11:06.973 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-29T04:11:07.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.099+0000 7f1ac67b5640 1 Processor -- start 2024-03-29T04:11:07.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.099+0000 7f1ac67b5640 1 -- start start 2024-03-29T04:11:07.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac67b5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1ac0156a80 0x7f1ac0071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac67b5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1ac00cf5c0 0x7f1ac015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac67b5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac67b5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac006ee90 con 0x7f1ac0156a80 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac67b5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac015d840 con 0x7f1ac00cf5c0 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac67b5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac00cfb60 con 0x7f1ac00d0470 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1ac0156a80 0x7f1ac0071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1ac0156a80 0x7f1ac0071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36218/0 (socket says 172.21.15.156:36218) 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 -- 172.21.15.156:0/3247385382 learned_addr learned my addr 172.21.15.156:0/3247385382 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1ac4d52640 1 --2- 172.21.15.156:0/3247385382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abf7fe640 1 --2- 172.21.15.156:0/3247385382 >> v2:172.21.15.203:3300/0 conn(0x7f1ac00cf5c0 0x7f1ac015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 -- 172.21.15.156:0/3247385382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 msgr2=0x7f1ac00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 --2- 172.21.15.156:0/3247385382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 -- 172.21.15.156:0/3247385382 >> v2:172.21.15.203:3300/0 conn(0x7f1ac00cf5c0 msgr2=0x7f1ac015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 --2- 172.21.15.156:0/3247385382 >> v2:172.21.15.203:3300/0 conn(0x7f1ac00cf5c0 0x7f1ac015d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.100+0000 7f1abffff640 1 -- 172.21.15.156:0/3247385382 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac006f0e0 con 0x7f1ac0156a80 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.101+0000 7f1abffff640 1 --2- 172.21.15.156:0/3247385382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac0071c20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00b2dd0 tx=0x7f1ab002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26caa5b7e961ba47 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.101+0000 7f1abeffd640 1 -- 172.21.15.156:0/3247385382 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab002e070 con 0x7f1ac0156a80 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.101+0000 7f1abeffd640 1 -- 172.21.15.156:0/3247385382 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab0037070 con 0x7f1ac0156a80 2024-03-29T04:11:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.101+0000 7f1abeffd640 1 -- 172.21.15.156:0/3247385382 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0031a00 con 0x7f1ac0156a80 2024-03-29T04:11:07.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 -- 172.21.15.156:0/3247385382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 msgr2=0x7f1ac0071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/3247385382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac0071c20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00b2dd0 tx=0x7f1ab002b880 comp rx=0 tx=0).stop 2024-03-29T04:11:07.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 -- 172.21.15.156:0/3247385382 shutdown_connections 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/3247385382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/3247385382 >> v2:172.21.15.203:3300/0 conn(0x7f1ac00cf5c0 0x7f1ac015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/3247385382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac0071c20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 -- 172.21.15.156:0/3247385382 >> 172.21.15.156:0/3247385382 conn(0x7f1ac00b21e0 msgr2=0x7f1ac00b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 -- 172.21.15.156:0/3247385382 shutdown_connections 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.102+0000 7f1ac67b5640 1 -- 172.21.15.156:0/3247385382 wait complete. 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.103+0000 7f1ac67b5640 1 Processor -- start 2024-03-29T04:11:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.103+0000 7f1ac67b5640 1 -- start start 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac67b5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac00cf5c0 0x7f1ac0099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac67b5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac67b5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac01e3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac67b5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac006dd10 con 0x7f1ac0156a80 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac67b5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac006db90 con 0x7f1ac00cf5c0 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac67b5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac006de90 con 0x7f1ac00d0470 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac01e3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac01e3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36232/0 (socket says 172.21.15.156:36232) 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 -- 172.21.15.156:0/1697128825 learned_addr learned my addr 172.21.15.156:0/1697128825 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:07.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1abffff640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac00cf5c0 0x7f1ac0099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1abf7fe640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 -- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 msgr2=0x7f1ac00b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac00d0470 0x7f1ac00b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 -- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac00cf5c0 msgr2=0x7f1ac0099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac00cf5c0 0x7f1ac0099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 -- 172.21.15.156:0/1697128825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac00b0ec0 con 0x7f1ac0156a80 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1abffff640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac00cf5c0 0x7f1ac0099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.104+0000 7f1ac4d52640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac01e3630 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8007e70 tx=0x7f1ab800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.105+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8014620 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.105+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab8019070 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.105+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8014a20 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.105+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac00b11a0 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.105+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac00b1760 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.106+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ac0073a40 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.106+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1ab8014bc0 con 0x7f1ac0156a80 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.106+0000 7f1abcff9640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1aac06c530 0x7f1aac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f1ab808bc10 con 0x7f1ac0156a80 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abffff640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1aac06c530 0x7f1aac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abcff9640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f1aac0723f0 0x7f1aac074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 --> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ab800ce50 con 0x7f1aac0723f0 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1ab808e310 con 0x7f1ac0156a80 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abf7fe640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f1aac0723f0 0x7f1aac074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abffff640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1aac06c530 0x7f1aac06e9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036000 tx=0x7f1ab0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.107+0000 7f1abf7fe640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f1aac0723f0 0x7f1aac074810 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.108+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== osd.7 v2:172.21.15.203:6817/3097526350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ab800ce50 con 0x7f1aac0723f0 2024-03-29T04:11:07.100 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.121+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 --> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1ac015a180 con 0x7f1aac0723f0 2024-03-29T04:11:07.100 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1abcff9640 1 -- 172.21.15.156:0/1697128825 <== osd.7 v2:172.21.15.203:6817/3097526350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1ac015a180 con 0x7f1aac0723f0 2024-03-29T04:11:07.101 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-29T04:11:07.101 INFO:teuthology.orchestra.run.smithi156.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T04:11:07.101 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-29T04:11:07.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f1aac0723f0 msgr2=0x7f1aac074810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.203:6817/3097526350,v1:172.21.15.203:6819/3097526350] conn(0x7f1aac0723f0 0x7f1aac074810 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1aac06c530 msgr2=0x7f1aac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f1aac06c530 0x7f1aac06e9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036000 tx=0x7f1ab0008030 comp rx=0 tx=0).stop 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 msgr2=0x7f1ac01e3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 --2- 172.21.15.156:0/1697128825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0156a80 0x7f1ac01e3630 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8007e70 tx=0x7f1ab800e4b0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac4d52640 1 -- 172.21.15.156:0/1697128825 reap_dead start 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 shutdown_connections 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 >> 172.21.15.156:0/1697128825 conn(0x7f1ac00b21e0 msgr2=0x7f1ac0071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 shutdown_connections 2024-03-29T04:11:07.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.122+0000 7f1ac67b5640 1 -- 172.21.15.156:0/1697128825 wait complete. 2024-03-29T04:11:07.112 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-29T04:11:07.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.234+0000 7f60b373d640 1 Processor -- start 2024-03-29T04:11:07.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.234+0000 7f60b373d640 1 -- start start 2024-03-29T04:11:07.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b373d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0af6b0 0x7f60ac0b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b373d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f60ac15c990 0x7f60ac15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b373d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f60ac0d0280 0x7f60ac0d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b373d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f60ac0b3050 con 0x7f60ac15c990 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b373d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f60ac0afa90 con 0x7f60ac0d0280 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b373d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f60ac15d510 con 0x7f60ac0af6b0 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b0cd8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f60ac15c990 0x7f60ac15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b0cd8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f60ac15c990 0x7f60ac15ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36240/0 (socket says 172.21.15.156:36240) 2024-03-29T04:11:07.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b0cd8640 1 -- 172.21.15.156:0/2611118479 learned_addr learned my addr 172.21.15.156:0/2611118479 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b1cda640 1 --2- 172.21.15.156:0/2611118479 >> v2:172.21.15.203:3300/0 conn(0x7f60ac0d0280 0x7f60ac0d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.235+0000 7f60b14d9640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0af6b0 0x7f60ac0b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b0cd8640 1 -- 172.21.15.156:0/2611118479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0af6b0 msgr2=0x7f60ac0b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b0cd8640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0af6b0 0x7f60ac0b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b0cd8640 1 -- 172.21.15.156:0/2611118479 >> v2:172.21.15.203:3300/0 conn(0x7f60ac0d0280 msgr2=0x7f60ac0d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b0cd8640 1 --2- 172.21.15.156:0/2611118479 >> v2:172.21.15.203:3300/0 conn(0x7f60ac0d0280 0x7f60ac0d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b0cd8640 1 -- 172.21.15.156:0/2611118479 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac15d7f0 con 0x7f60ac15c990 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b14d9640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0af6b0 0x7f60ac0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:07.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.236+0000 7f60b0cd8640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac15c990 0x7f60ac15ce10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0b5020 tx=0x7f609c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5bc70ebf66502e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60a3fff640 1 -- 172.21.15.156:0/2611118479 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c035020 con 0x7f60ac15c990 2024-03-29T04:11:07.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60a3fff640 1 -- 172.21.15.156:0/2611118479 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c031870 con 0x7f60ac15c990 2024-03-29T04:11:07.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60a3fff640 1 -- 172.21.15.156:0/2611118479 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c031c40 con 0x7f60ac15c990 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 -- 172.21.15.156:0/2611118479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac15c990 msgr2=0x7f60ac15ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac15c990 0x7f60ac15ce10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0b5020 tx=0x7f609c02b910 comp rx=0 tx=0).stop 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 -- 172.21.15.156:0/2611118479 shutdown_connections 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac15c990 0x7f60ac15ce10 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 --2- 172.21.15.156:0/2611118479 >> v2:172.21.15.203:3300/0 conn(0x7f60ac0d0280 0x7f60ac0d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 --2- 172.21.15.156:0/2611118479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0af6b0 0x7f60ac0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 -- 172.21.15.156:0/2611118479 >> 172.21.15.156:0/2611118479 conn(0x7f60ac0b1f30 msgr2=0x7f60ac0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:07.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 -- 172.21.15.156:0/2611118479 shutdown_connections 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.237+0000 7f60b373d640 1 -- 172.21.15.156:0/2611118479 wait complete. 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.238+0000 7f60b373d640 1 Processor -- start 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.238+0000 7f60b373d640 1 -- start start 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b373d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 0x7f60ac099840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b373d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0d0280 0x7f60ac09c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b373d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ac15c990 0x7f60ac09cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b373d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac0ad200 con 0x7f60ac0af6b0 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b373d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac0b35f0 con 0x7f60ac15c990 2024-03-29T04:11:07.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b373d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f60ac15dad0 con 0x7f60ac0d0280 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b14d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 0x7f60ac099840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b14d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 0x7f60ac099840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36256/0 (socket says 172.21.15.156:36256) 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b14d9640 1 -- 172.21.15.156:0/2893485410 learned_addr learned my addr 172.21.15.156:0/2893485410 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b1cda640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ac15c990 0x7f60ac09cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b0cd8640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0d0280 0x7f60ac09c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.239+0000 7f60b14d9640 1 -- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0d0280 msgr2=0x7f60ac09c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b14d9640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0d0280 0x7f60ac09c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b14d9640 1 -- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ac15c990 msgr2=0x7f60ac09cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.223 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b14d9640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ac15c990 0x7f60ac09cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b14d9640 1 -- 172.21.15.156:0/2893485410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac0cdec0 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b14d9640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 0x7f60ac099840 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e7e0 tx=0x7f60a800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a800f040 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a8004030 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ac0ce1a0 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8004400 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.240+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ac0ce760 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.241+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ac09e3b0 con 0x7f60ac0af6b0 2024-03-29T04:11:07.224 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.241+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f60a800c6d0 con 0x7f60ac0af6b0 2024-03-29T04:11:07.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.242+0000 7f60a1ffb640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f609006c530 0x7f609006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.242+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f60a808b0e0 con 0x7f60ac0af6b0 2024-03-29T04:11:07.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.242+0000 7f60b0cd8640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f609006c530 0x7f609006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.242+0000 7f60b0cd8640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f609006c530 0x7f609006e9f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f609c036000 tx=0x7f609c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.225 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.246+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f60a8017020 con 0x7f60ac0af6b0 2024-03-29T04:11:07.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.421+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f60ac0b4d20 con 0x7f609006c530 2024-03-29T04:11:07.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.421+0000 7f60a1ffb640 1 -- 172.21.15.156:0/2893485410 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f60ac0b4d20 con 0x7f609006c530 2024-03-29T04:11:07.401 INFO:teuthology.orchestra.run.smithi156.stderr:instructed osd(s) 7 to deep-scrub 2024-03-29T04:11:07.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f609006c530 msgr2=0x7f609006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f609006c530 0x7f609006e9f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f609c036000 tx=0x7f609c03c040 comp rx=0 tx=0).stop 2024-03-29T04:11:07.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 msgr2=0x7f60ac099840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 0x7f60ac099840 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e7e0 tx=0x7f60a800eca0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 shutdown_connections 2024-03-29T04:11:07.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f609006c530 0x7f609006e9f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ac15c990 0x7f60ac09cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60ac0d0280 0x7f60ac09c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 --2- 172.21.15.156:0/2893485410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac0af6b0 0x7f60ac099840 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 >> 172.21.15.156:0/2893485410 conn(0x7f60ac0b1f30 msgr2=0x7f60ac15ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:07.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 shutdown_connections 2024-03-29T04:11:07.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.423+0000 7f60b373d640 1 -- 172.21.15.156:0/2893485410 wait complete. 2024-03-29T04:11:07.413 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:11:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.534+0000 7f6e94ace640 1 Processor -- start 2024-03-29T04:11:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.534+0000 7f6e94ace640 1 -- start start 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.535+0000 7f6e94ace640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6e900b3050 0x7f6e9006f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.535+0000 7f6e94ace640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6e9006f5d0 0x7f6e90071750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.535+0000 7f6e94ace640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e900afa90 con 0x7f6e900af6b0 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.535+0000 7f6e94ace640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6e900b3510 con 0x7f6e9006f5d0 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.535+0000 7f6e94ace640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6e900b37f0 con 0x7f6e900b3050 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.535+0000 7f6e8ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6e9006f5d0 0x7f6e90071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6e9006f5d0 0x7f6e90071750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36268/0 (socket says 172.21.15.156:36268) 2024-03-29T04:11:07.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 -- 172.21.15.156:0/453717369 learned_addr learned my addr 172.21.15.156:0/453717369 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8dd9b640 1 --2- 172.21.15.156:0/453717369 >> v2:172.21.15.203:3300/0 conn(0x7f6e900b3050 0x7f6e9006f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 -- 172.21.15.156:0/453717369 >> v2:172.21.15.203:3300/0 conn(0x7f6e900b3050 msgr2=0x7f6e9006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 --2- 172.21.15.156:0/453717369 >> v2:172.21.15.203:3300/0 conn(0x7f6e900b3050 0x7f6e9006f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 -- 172.21.15.156:0/453717369 >> v1:172.21.15.203:6789/0 conn(0x7f6e900af6b0 legacy=0x7f6e9015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 -- 172.21.15.156:0/453717369 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e900b3ad0 con 0x7f6e9006f5d0 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.536+0000 7f6e8ed9d640 1 --2- 172.21.15.156:0/453717369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e90071750 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6e90075030 tx=0x7f6e8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb27529d38cc764 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e8d59a640 1 -- 172.21.15.156:0/453717369 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8802e070 con 0x7f6e9006f5d0 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e8d59a640 1 -- 172.21.15.156:0/453717369 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e88037070 con 0x7f6e9006f5d0 2024-03-29T04:11:07.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e8d59a640 1 -- 172.21.15.156:0/453717369 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88031a00 con 0x7f6e9006f5d0 2024-03-29T04:11:07.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 -- 172.21.15.156:0/453717369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 msgr2=0x7f6e90071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 --2- 172.21.15.156:0/453717369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e90071750 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6e90075030 tx=0x7f6e8802b880 comp rx=0 tx=0).stop 2024-03-29T04:11:07.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 -- 172.21.15.156:0/453717369 shutdown_connections 2024-03-29T04:11:07.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 --2- 172.21.15.156:0/453717369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e90071750 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 --2- 172.21.15.156:0/453717369 >> v2:172.21.15.203:3300/0 conn(0x7f6e900b3050 0x7f6e9006f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 -- 172.21.15.156:0/453717369 >> 172.21.15.156:0/453717369 conn(0x7f6e900b1d50 msgr2=0x7f6e900b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 -- 172.21.15.156:0/453717369 shutdown_connections 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.537+0000 7f6e94ace640 1 -- 172.21.15.156:0/453717369 wait complete. 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 Processor -- start 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 -- start start 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e901f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e900af6b0 0x7f6e900ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e900b3050 0x7f6e9009d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6e900b45b0 con 0x7f6e9006f5d0 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e900b4090 con 0x7f6e900b3050 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e94ace640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f6e90071c90 con 0x7f6e900af6b0 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e8e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e901f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e8e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e901f2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36276/0 (socket says 172.21.15.156:36276) 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.539+0000 7f6e8e59c640 1 -- 172.21.15.156:0/1120797091 learned_addr learned my addr 172.21.15.156:0/1120797091 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8dd9b640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e900af6b0 0x7f6e900ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8ed9d640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e900b3050 0x7f6e9009d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8e59c640 1 -- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e900af6b0 msgr2=0x7f6e900ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8e59c640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e900af6b0 0x7f6e900ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8e59c640 1 -- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e900b3050 msgr2=0x7f6e9009d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8e59c640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e900b3050 0x7f6e9009d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8e59c640 1 -- 172.21.15.156:0/1120797091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e901f27d0 con 0x7f6e9006f5d0 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8dd9b640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e900af6b0 0x7f6e900ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e8e59c640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e901f2180 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000c7f0 tx=0x7f6e8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80016020 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8000aa50 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e901f2ab0 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8000ae40 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.540+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e901f3070 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.541+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e90068610 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.541+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6e80019020 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.542+0000 7f6e7effd640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e7006c530 0x7f6e7006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.542+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f6e8008b070 con 0x7f6e9006f5d0 2024-03-29T04:11:07.525 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.542+0000 7f6e8dd9b640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e7006c530 0x7f6e7006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:07.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.543+0000 7f6e8dd9b640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e7006c530 0x7f6e7006e9f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6e84006fd0 tx=0x7f6e84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:07.526 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.546+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6e80020070 con 0x7f6e9006f5d0 2024-03-29T04:11:07.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.721+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6e9009dbb0 con 0x7f6e7006c530 2024-03-29T04:11:07.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.722+0000 7f6e7effd640 1 -- 172.21.15.156:0/1120797091 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7f6e9009dbb0 con 0x7f6e7006c530 2024-03-29T04:11:07.702 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:07.702 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e7006c530 msgr2=0x7f6e7006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e7006c530 0x7f6e7006e9f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6e84006fd0 tx=0x7f6e84008040 comp rx=0 tx=0).stop 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 msgr2=0x7f6e901f2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e901f2180 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000c7f0 tx=0x7f6e8000ccb0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 shutdown_connections 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f6e7006c530 0x7f6e7006e9f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.704 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e900b3050 0x7f6e9009d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e900af6b0 0x7f6e900ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 --2- 172.21.15.156:0/1120797091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e9006f5d0 0x7f6e901f2180 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:07.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 >> 172.21.15.156:0/1120797091 conn(0x7f6e900b1d50 msgr2=0x7f6e900afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:07.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 shutdown_connections 2024-03-29T04:11:07.705 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:07.724+0000 7f6e94ace640 1 -- 172.21.15.156:0/1120797091 wait complete. 2024-03-29T04:11:07.715 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-03-29T04:11:07.661040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":237296,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737424,"statfs":{"total":773094113280,"available":772851122176,"internally_reserved":0,"allocated":4595712,"data_stored":2523200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-173890432,"num_objects":-116,"num_object_clones":0,"num_object_copies":-232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-116,"num_whiteouts":0,"num_read":-41,"num_read_kb":0,"num_write":-175,"num_write_kb":-169853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.155446"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-29T04:11:00.647049+0000","last_change":"2024-03-29T04:08:43.001496+0000","last_active":"2024-03-29T04:11:00.647049+0000","last_peered":"2024-03-29T04:11:00.647049+0000","last_clean":"2024-03-29T04:11:00.647049+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:11:00.647049+0000","last_undegraded":"2024-03-29T04:11:00.647049+0000","last_fullsized":"2024-03-29T04:11:00.647049+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T04:08:41.750308+0000","last_clean_scrub_stamp":"2024-03-29T04:08:41.750308+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:48:21.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542176,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1019904,"data_stored":758176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31600,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":758176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:11:07.718 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-29T04:08:41.750308+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=29, tm_hour=4, tm_min=8, tm_sec=41, tm_wday=4, tm_yday=89, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=29, tm_hour=4, tm_min=11, tm_sec=2, tm_wday=4, tm_yday=89, tm_isdst=0) 2024-03-29T04:11:07.718 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-29T04:11:26.157 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:11:26.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:11:27.719 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T04:11:27.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.851+0000 7f3ef57db640 1 Processor -- start 2024-03-29T04:11:27.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.851+0000 7f3ef57db640 1 -- start start 2024-03-29T04:11:27.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3ef57db640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3ef00d42c0 0x7f3ef00d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3ef57db640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ef006cbb0 0x7f3ef006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3ef57db640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef00d4c80 con 0x7f3ef015e730 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3ef57db640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3ef006c720 con 0x7f3ef006cbb0 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3ef57db640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3ef00cfa20 con 0x7f3ef00d42c0 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3eef7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ef006cbb0 0x7f3ef006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3eef7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ef006cbb0 0x7f3ef006cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36998/0 (socket says 172.21.15.156:36998) 2024-03-29T04:11:27.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.852+0000 7f3eef7fe640 1 -- 172.21.15.156:0/2250167406 learned_addr learned my addr 172.21.15.156:0/2250167406 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.853+0000 7f3eee7fc640 1 --2- 172.21.15.156:0/2250167406 >> v2:172.21.15.203:3300/0 conn(0x7f3ef00d42c0 0x7f3ef00d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.853+0000 7f3eef7fe640 1 -- 172.21.15.156:0/2250167406 >> v2:172.21.15.203:3300/0 conn(0x7f3ef00d42c0 msgr2=0x7f3ef00d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.853+0000 7f3eef7fe640 1 --2- 172.21.15.156:0/2250167406 >> v2:172.21.15.203:3300/0 conn(0x7f3ef00d42c0 0x7f3ef00d4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.853+0000 7f3eef7fe640 1 -- 172.21.15.156:0/2250167406 >> v1:172.21.15.203:6789/0 conn(0x7f3ef015e730 legacy=0x7f3ef00ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.853+0000 7f3eef7fe640 1 -- 172.21.15.156:0/2250167406 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef00cfd00 con 0x7f3ef006cbb0 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.853+0000 7f3eef7fe640 1 --2- 172.21.15.156:0/2250167406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef006cbb0 0x7f3ef006cfe0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00ba4e0 tx=0x7f3ee802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bff69a1cae3d3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:27.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3eedffb640 1 -- 172.21.15.156:0/2250167406 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8034070 con 0x7f3ef006cbb0 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3eedffb640 1 -- 172.21.15.156:0/2250167406 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee8030860 con 0x7f3ef006cbb0 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3eedffb640 1 -- 172.21.15.156:0/2250167406 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8030c30 con 0x7f3ef006cbb0 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 -- 172.21.15.156:0/2250167406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef006cbb0 msgr2=0x7f3ef006cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 --2- 172.21.15.156:0/2250167406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef006cbb0 0x7f3ef006cfe0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00ba4e0 tx=0x7f3ee802b9a0 comp rx=0 tx=0).stop 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 -- 172.21.15.156:0/2250167406 shutdown_connections 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 --2- 172.21.15.156:0/2250167406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef006cbb0 0x7f3ef006cfe0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 --2- 172.21.15.156:0/2250167406 >> v2:172.21.15.203:3300/0 conn(0x7f3ef00d42c0 0x7f3ef00d4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:27.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 -- 172.21.15.156:0/2250167406 >> 172.21.15.156:0/2250167406 conn(0x7f3ef008bfd0 msgr2=0x7f3ef009d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:27.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 -- 172.21.15.156:0/2250167406 shutdown_connections 2024-03-29T04:11:27.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.854+0000 7f3ef57db640 1 -- 172.21.15.156:0/2250167406 wait complete. 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 Processor -- start 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 -- start start 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef006cbb0 0x7f3ef01603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ef00d42c0 0x7f3ef0153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 0x7f3ef015a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef00d2220 con 0x7f3ef015e730 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef00d02c0 con 0x7f3ef006cbb0 2024-03-29T04:11:27.841 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3ef57db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef015eb10 con 0x7f3ef00d42c0 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 0x7f3ef015a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 0x7f3ef015a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37014/0 (socket says 172.21.15.156:37014) 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.856+0000 7f3eef7fe640 1 -- 172.21.15.156:0/3203473516 learned_addr learned my addr 172.21.15.156:0/3203473516 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eeeffd640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef006cbb0 0x7f3ef01603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eee7fc640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ef00d42c0 0x7f3ef0153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eef7fe640 1 -- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ef00d42c0 msgr2=0x7f3ef0153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eef7fe640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ef00d42c0 0x7f3ef0153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:27.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eef7fe640 1 -- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef006cbb0 msgr2=0x7f3ef01603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eef7fe640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef006cbb0 0x7f3ef01603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eef7fe640 1 -- 172.21.15.156:0/3203473516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef01608f0 con 0x7f3ef015e730 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eeeffd640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef006cbb0 0x7f3ef01603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eee7fc640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ef00d42c0 0x7f3ef0153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3eef7fe640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 0x7f3ef015a370 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3ee802b9e0 tx=0x7f3ee8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8048070 con 0x7f3ef015e730 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee8030660 con 0x7f3ef015e730 2024-03-29T04:11:27.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef015fdf0 con 0x7f3ef015e730 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8034040 con 0x7f3ef015e730 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.857+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef00c8fc0 con 0x7f3ef015e730 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.858+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef0068610 con 0x7f3ef015e730 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.859+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3ee804b070 con 0x7f3ef015e730 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.859+0000 7f3ed77fe640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3edc06c560 0x7f3edc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.859+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f3ee80ad510 con 0x7f3ef015e730 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.859+0000 7f3eeeffd640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3edc06c560 0x7f3edc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:27.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.860+0000 7f3eeeffd640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3edc06c560 0x7f3edc06ea20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0001d60 tx=0x7f3ee0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:27.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:27.864+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3ef015fdf0 con 0x7f3ef015e730 2024-03-29T04:11:28.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.036+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 --> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ef00b2b90 con 0x7f3edc06c560 2024-03-29T04:11:28.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.037+0000 7f3ed77fe640 1 -- 172.21.15.156:0/3203473516 <== mgr.4107 v2:172.21.15.156:6832/2875364846 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13357 (secure 0 0 0) 0x7f3ef00b2b90 con 0x7f3edc06c560 2024-03-29T04:11:28.017 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:28.018 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-03-29T04:11:28.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3edc06c560 msgr2=0x7f3edc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3edc06c560 0x7f3edc06ea20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0001d60 tx=0x7f3ee0009040 comp rx=0 tx=0).stop 2024-03-29T04:11:28.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 msgr2=0x7f3ef015a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 0x7f3ef015a370 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3ee802b9e0 tx=0x7f3ee8004730 comp rx=0 tx=0).stop 2024-03-29T04:11:28.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 shutdown_connections 2024-03-29T04:11:28.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f3edc06c560 0x7f3edc06ea20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ef015e730 0x7f3ef015a370 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ef00d42c0 0x7f3ef0153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 --2- 172.21.15.156:0/3203473516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef006cbb0 0x7f3ef01603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 >> 172.21.15.156:0/3203473516 conn(0x7f3ef008bfd0 msgr2=0x7f3ef00b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 shutdown_connections 2024-03-29T04:11:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.040+0000 7f3ef57db640 1 -- 172.21.15.156:0/3203473516 wait complete. 2024-03-29T04:11:28.031 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-03-29T04:11:27.665009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":237288,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737432,"statfs":{"total":773094113280,"available":772851130368,"internally_reserved":0,"allocated":4595712,"data_stored":2523200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002434"},"pg_stats":[{"pgid":"1.0","version":"37'119","reported_seq":211,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-29T04:11:07.995923+0000","last_change":"2024-03-29T04:11:07.995983+0000","last_active":"2024-03-29T04:11:07.995923+0000","last_peered":"2024-03-29T04:11:07.995923+0000","last_clean":"2024-03-29T04:11:07.995923+0000","last_became_active":"2024-03-29T04:08:43.000457+0000","last_became_peered":"2024-03-29T04:08:43.000457+0000","last_unstale":"2024-03-29T04:11:07.995923+0000","last_undegraded":"2024-03-29T04:11:07.995923+0000","last_fullsized":"2024-03-29T04:11:07.995923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'119","last_scrub_stamp":"2024-03-29T04:11:07.995809+0000","last_deep_scrub":"37'119","last_deep_scrub_stamp":"2024-03-29T04:11:07.995809+0000","last_clean_scrub_stamp":"2024-03-29T04:11:07.995809+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:44.963105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1019904,"data_stored":758176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31600,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1019904,"data_stored":758176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T04:11:28.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-29T04:11:28.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.152+0000 7f798a8c5640 1 Processor -- start 2024-03-29T04:11:28.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.152+0000 7f798a8c5640 1 -- start start 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f798a8c5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 0x7f7984072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f798a8c5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f798406e670 0x7f798415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f798a8c5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79840727d0 con 0x7f79840b3050 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f798a8c5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f798415f4d0 con 0x7f798406e670 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f798a8c5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f7984072ab0 con 0x7f798406e290 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f79837fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f798406e670 0x7f798415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f7983fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 0x7f7984072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f7988e62640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f79840b3050 0x7f79840b34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.156:59660/0 (socket says 172.21.15.156:59660) 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f79837fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f798406e670 0x7f798415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.156:45788/0 (socket says 172.21.15.156:45788) 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f7983fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 0x7f7984072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.156:47444/0 (socket says 172.21.15.156:47444) 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f7988e62640 1 -- 172.21.15.156:0/2624747389 learned_addr learned my addr 172.21.15.156:0/2624747389 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f79837fe640 1 -- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 msgr2=0x7f7984072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f79837fe640 1 --2- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 0x7f7984072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f79837fe640 1 -- 172.21.15.156:0/2624747389 >> v1:172.21.15.203:6789/0 conn(0x7f79840b3050 legacy=0x7f79840b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f79837fe640 1 -- 172.21.15.156:0/2624747389 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7984072d90 con 0x7f798406e670 2024-03-29T04:11:28.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.153+0000 7f7983fff640 1 --2- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 0x7f7984072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T04:11:28.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.154+0000 7f79837fe640 1 --2- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e670 0x7f798415ef90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f798415f650 tx=0x7f797402ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ca220dc5dcc424 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:28.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.154+0000 7f7982ffd640 1 -- 172.21.15.156:0/2624747389 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797403a070 con 0x7f798406e670 2024-03-29T04:11:28.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.154+0000 7f7982ffd640 1 -- 172.21.15.156:0/2624747389 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f797402daa0 con 0x7f798406e670 2024-03-29T04:11:28.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.154+0000 7f7982ffd640 1 -- 172.21.15.156:0/2624747389 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797400b040 con 0x7f798406e670 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 -- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e670 msgr2=0x7f798415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 --2- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e670 0x7f798415ef90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f798415f650 tx=0x7f797402ccf0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 -- 172.21.15.156:0/2624747389 shutdown_connections 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 --2- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e670 0x7f798415ef90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 --2- 172.21.15.156:0/2624747389 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e290 0x7f7984072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 -- 172.21.15.156:0/2624747389 >> 172.21.15.156:0/2624747389 conn(0x7f79840b1f30 msgr2=0x7f79840b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 -- 172.21.15.156:0/2624747389 shutdown_connections 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.155+0000 7f798a8c5640 1 -- 172.21.15.156:0/2624747389 wait complete. 2024-03-29T04:11:28.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.156+0000 7f798a8c5640 1 Processor -- start 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.156+0000 7f798a8c5640 1 -- start start 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f798a8c5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e290 0x7f7984153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f798a8c5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e670 0x7f79841536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f798a8c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 0x7f79841f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f798a8c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79840739f0 con 0x7f79840b3050 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f798a8c5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7984073870 con 0x7f798406e290 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f798a8c5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f79840b3bd0 con 0x7f798406e670 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f7988e62640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 0x7f79841f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f7988e62640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 0x7f79841f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37028/0 (socket says 172.21.15.156:37028) 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f7988e62640 1 -- 172.21.15.156:0/3271682592 learned_addr learned my addr 172.21.15.156:0/3271682592 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f7983fff640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e290 0x7f7984153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.157+0000 7f79837fe640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e670 0x7f79841536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7988e62640 1 -- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e670 msgr2=0x7f79841536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7988e62640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e670 0x7f79841536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7988e62640 1 -- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e290 msgr2=0x7f7984153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7988e62640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e290 0x7f7984153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7988e62640 1 -- 172.21.15.156:0/3271682592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79841520f0 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7988e62640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 0x7f79841f2750 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f797c00ca80 tx=0x7f797c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c017070 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f797c00ac30 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79841523d0 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c012600 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.158+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7984152990 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.159+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798409e3b0 con 0x7f79840b3050 2024-03-29T04:11:28.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.159+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f797c01a070 con 0x7f79840b3050 2024-03-29T04:11:28.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.160+0000 7f7980ff9640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f797006c530 0x7f797006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.160+0000 7f7983fff640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f797006c530 0x7f797006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.160+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f797c08afb0 con 0x7f79840b3050 2024-03-29T04:11:28.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.160+0000 7f7983fff640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f797006c530 0x7f797006e9f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7978009560 tx=0x7f7978009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:28.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.165+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f797c00ae20 con 0x7f79840b3050 2024-03-29T04:11:28.320 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.340+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f7984071cf0 con 0x7f79840b3050 2024-03-29T04:11:28.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.343+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f797c059310 con 0x7f79840b3050 2024-03-29T04:11:28.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.344+0000 7f7980ff9640 1 -- 172.21.15.156:0/3271682592 <== mon.0 v2:172.21.15.156:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f797c05d610 con 0x7f79840b3050 2024-03-29T04:11:28.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f797006c530 msgr2=0x7f797006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f797006c530 0x7f797006e9f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7978009560 tx=0x7f7978009190 comp rx=0 tx=0).stop 2024-03-29T04:11:28.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 msgr2=0x7f79841f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 0x7f79841f2750 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f797c00ca80 tx=0x7f797c00cf40 comp rx=0 tx=0).stop 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 shutdown_connections 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f797006c530 0x7f797006e9f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79840b3050 0x7f79841f2750 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f798406e670 0x7f79841536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 --2- 172.21.15.156:0/3271682592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f798406e290 0x7f7984153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 >> 172.21.15.156:0/3271682592 conn(0x7f79840b1f30 msgr2=0x7f79840afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 shutdown_connections 2024-03-29T04:11:28.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.346+0000 7f798a8c5640 1 -- 172.21.15.156:0/3271682592 wait complete. 2024-03-29T04:11:28.337 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T04:11:28.338 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T04:11:28.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 Processor -- start 2024-03-29T04:11:28.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 -- start start 2024-03-29T04:11:28.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0e340b3050 0x7f0e3406f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0e3406f5d0 0x7f0e34071750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e340afa90 con 0x7f0e340af6b0 2024-03-29T04:11:28.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f0e340b3510 con 0x7f0e3406f5d0 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.459+0000 7f0e3b1db640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0e340b37f0 con 0x7f0e340b3050 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0e3406f5d0 0x7f0e34071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0e3406f5d0 0x7f0e34071750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37040/0 (socket says 172.21.15.156:37040) 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 -- 172.21.15.156:0/4187488214 learned_addr learned my addr 172.21.15.156:0/4187488214 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e38f77640 1 --1- 172.21.15.156:0/4187488214 >> v1:172.21.15.156:6789/0 conn(0x7f0e340af6b0 0x7f0e3415c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:52382/0 (socket says 172.21.15.156:52382) 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e33fff640 1 --2- 172.21.15.156:0/4187488214 >> v2:172.21.15.203:3300/0 conn(0x7f0e340b3050 0x7f0e3406f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 -- 172.21.15.156:0/4187488214 >> v2:172.21.15.203:3300/0 conn(0x7f0e340b3050 msgr2=0x7f0e3406f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T04:11:28.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 --2- 172.21.15.156:0/4187488214 >> v2:172.21.15.203:3300/0 conn(0x7f0e340b3050 0x7f0e3406f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 -- 172.21.15.156:0/4187488214 >> v1:172.21.15.156:6789/0 conn(0x7f0e340af6b0 legacy=0x7f0e3415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e337fe640 1 -- 172.21.15.156:0/4187488214 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4151966141 0 0) 0x7f0e340afa90 con 0x7f0e340af6b0 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.460+0000 7f0e39778640 1 -- 172.21.15.156:0/4187488214 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e340b3ad0 con 0x7f0e3406f5d0 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e39778640 1 --2- 172.21.15.156:0/4187488214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3406f5d0 0x7f0e34071750 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0e34075030 tx=0x7f0e2c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ee1b8af6edcc15 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e337fe640 1 -- 172.21.15.156:0/4187488214 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c02ca80 con 0x7f0e3406f5d0 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e337fe640 1 -- 172.21.15.156:0/4187488214 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0e2c02cc20 con 0x7f0e3406f5d0 2024-03-29T04:11:28.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e337fe640 1 -- 172.21.15.156:0/4187488214 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0343f0 con 0x7f0e3406f5d0 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e3b1db640 1 -- 172.21.15.156:0/4187488214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3406f5d0 msgr2=0x7f0e34071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/4187488214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3406f5d0 0x7f0e34071750 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0e34075030 tx=0x7f0e2c02bee0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e3b1db640 1 -- 172.21.15.156:0/4187488214 shutdown_connections 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/4187488214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e3406f5d0 0x7f0e34071750 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/4187488214 >> v2:172.21.15.203:3300/0 conn(0x7f0e340b3050 0x7f0e3406f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.461+0000 7f0e3b1db640 1 -- 172.21.15.156:0/4187488214 >> 172.21.15.156:0/4187488214 conn(0x7f0e340b1d50 msgr2=0x7f0e340b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.462+0000 7f0e3b1db640 1 -- 172.21.15.156:0/4187488214 shutdown_connections 2024-03-29T04:11:28.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.462+0000 7f0e3b1db640 1 -- 172.21.15.156:0/4187488214 wait complete. 2024-03-29T04:11:28.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.463+0000 7f0e3b1db640 1 Processor -- start 2024-03-29T04:11:28.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.463+0000 7f0e3b1db640 1 -- start start 2024-03-29T04:11:28.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.463+0000 7f0e3b1db640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e3406f5d0 0x7f0e340ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e340af6b0 0x7f0e341f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 0x7f0e341596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0e340b45b0 con 0x7f0e340b3050 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e340b4090 con 0x7f0e340af6b0 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 -- 0x7f0e34071c90 con 0x7f0e3406f5d0 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 0x7f0e341596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 0x7f0e341596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37044/0 (socket says 172.21.15.156:37044) 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 -- 172.21.15.156:0/3851070435 learned_addr learned my addr 172.21.15.156:0/3851070435 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e38f77640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e3406f5d0 0x7f0e340ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e33fff640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e340af6b0 0x7f0e341f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 -- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e3406f5d0 msgr2=0x7f0e340ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e3406f5d0 0x7f0e340ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 -- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e340af6b0 msgr2=0x7f0e341f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e340af6b0 0x7f0e341f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 -- 172.21.15.156:0/3851070435 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e340ce690 con 0x7f0e340b3050 2024-03-29T04:11:28.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e39778640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 0x7f0e341596e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c036040 tx=0x7f0e2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c043020 con 0x7f0e340b3050 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0e2c004250 con 0x7f0e340b3050 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e341f4cb0 con 0x7f0e340b3050 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c009510 con 0x7f0e340b3050 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.464+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e341f5270 con 0x7f0e340b3050 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.465+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e340b1ad0 con 0x7f0e340b3050 2024-03-29T04:11:28.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.466+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0e2c046020 con 0x7f0e340b3050 2024-03-29T04:11:28.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.466+0000 7f0e317fa640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0e2006c530 0x7f0e2006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T04:11:28.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.466+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f0e2c0ad930 con 0x7f0e340b3050 2024-03-29T04:11:28.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.466+0000 7f0e38f77640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0e2006c530 0x7f0e2006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T04:11:28.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.467+0000 7f0e38f77640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0e2006c530 0x7f0e2006e9f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0e24000f60 tx=0x7f0e24007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T04:11:28.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.471+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0e340b1ad0 con 0x7f0e340b3050 2024-03-29T04:11:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.682+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e34067df0 con 0x7f0e340b3050 2024-03-29T04:11:28.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.682+0000 7f0e317fa640 1 -- 172.21.15.156:0/3851070435 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3881 (secure 0 0 0) 0x7f0e34067df0 con 0x7f0e340b3050 2024-03-29T04:11:28.663 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-29T04:11:28.663 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2155154614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6833","nonce":2155154614},{"type":"v1","addr":"172.21.15.203:6835","nonce":2155154614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1399293419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1399293419},{"type":"v1","addr":"172.21.15.156:6836","nonce":1399293419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6834/1620163190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1620163190},{"type":"v1","addr":"172.21.15.203:6834","nonce":1620163190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5757,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1924333533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":1924333533},{"type":"v1","addr":"172.21.15.156:6837","nonce":1924333533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-03-29T04:11:28.663 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 19 2024-03-29T04:11:28.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0e2006c530 msgr2=0x7f0e2006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0e2006c530 0x7f0e2006e9f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0e24000f60 tx=0x7f0e24007d50 comp rx=0 tx=0).stop 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 msgr2=0x7f0e341596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 0x7f0e341596e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c036040 tx=0x7f0e2c004a40 comp rx=0 tx=0).stop 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 shutdown_connections 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:6832/2875364846,v1:172.21.15.156:6833/2875364846] conn(0x7f0e2006c530 0x7f0e2006e9f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e340b3050 0x7f0e341596e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e340af6b0 0x7f0e341f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 --2- 172.21.15.156:0/3851070435 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e3406f5d0 0x7f0e340ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T04:11:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 >> 172.21.15.156:0/3851070435 conn(0x7f0e340b1d50 msgr2=0x7f0e340cfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T04:11:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 shutdown_connections 2024-03-29T04:11:28.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-29T04:11:28.684+0000 7f0e3b1db640 1 -- 172.21.15.156:0/3851070435 wait complete. 2024-03-29T04:11:28.678 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-29T04:11:28.678 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-29T04:11:28.678 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T04:11:28.678 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.687 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T04:11:28.688 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T04:11:28.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.737 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T04:11:28.738 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T04:11:28.738 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.749 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T04:11:28.750 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T04:11:28.750 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.802 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T04:11:28.802 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T04:11:28.802 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T04:11:28.802 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.834 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T04:11:28.834 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T04:11:28.834 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.898 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T04:11:28.898 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T04:11:28.898 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:28.970 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T04:11:28.971 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T04:11:28.971 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.036 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T04:11:29.036 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T04:11:29.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.069 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T04:11:29.069 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T04:11:29.069 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.147 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T04:11:29.148 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T04:11:29.148 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.214 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T04:11:29.215 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T04:11:29.215 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.287 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T04:11:29.287 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T04:11:29.287 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T04:11:29.287 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.313 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T04:11:29.313 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T04:11:29.313 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.354 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T04:11:29.354 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T04:11:29.354 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T04:11:29.355 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.366 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T04:11:29.366 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T04:11:29.366 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.407 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T04:11:29.407 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T04:11:29.407 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T04:11:29.460 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T04:11:29.460 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T04:11:29.460 DEBUG:teuthology.orchestra.run.smithi156:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-29T04:11:29.494 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:11:29.494 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T04:11:29.725 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:11:29.725 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T04:11:29.788 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:11:29.789 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T04:11:29.839 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:11:29.839 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T04:11:29.890 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:11:29.890 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T04:11:30.064 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:11:30.064 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T04:11:30.136 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:11:30.136 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T04:11:30.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:11:30.194 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T04:11:30.269 INFO:tasks.ceph:Archiving mon data... 2024-03-29T04:11:30.270 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/data/mon.a.tgz 2024-03-29T04:11:30.270 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-29T04:11:30.286 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.MMRNWRWLyd 2024-03-29T04:11:30.286 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MMRNWRWLyd 2024-03-29T04:11:30.406 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.MMRNWRWLyd 2024-03-29T04:11:30.491 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.MMRNWRWLyd is 298KB 2024-03-29T04:11:30.586 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.MMRNWRWLyd 2024-03-29T04:11:30.603 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/data/mon.b.tgz 2024-03-29T04:11:30.603 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-29T04:11:30.620 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.pI77GG0ull 2024-03-29T04:11:30.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pI77GG0ull 2024-03-29T04:11:30.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.pI77GG0ull 2024-03-29T04:11:30.856 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.pI77GG0ull is 398KB 2024-03-29T04:11:30.954 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.pI77GG0ull 2024-03-29T04:11:30.968 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/data/mon.c.tgz 2024-03-29T04:11:30.969 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-29T04:11:31.023 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.z4o7oz2Lie 2024-03-29T04:11:31.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.z4o7oz2Lie 2024-03-29T04:11:31.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.z4o7oz2Lie 2024-03-29T04:11:31.248 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.z4o7oz2Lie is 398KB 2024-03-29T04:11:31.345 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.z4o7oz2Lie 2024-03-29T04:11:31.360 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T04:11:31.361 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T04:11:31.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T04:11:31.447 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T04:11:31.481 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T04:11:31.481 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T04:11:31.483 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/remote/smithi156/crash 2024-03-29T04:11:31.484 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T04:11:31.516 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/remote/smithi203/crash 2024-03-29T04:11:31.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T04:11:31.547 INFO:tasks.ceph:Shutting down logrotate 2024-03-29T04:11:31.548 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:11:31.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T04:11:31.638 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-29T04:11:31.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-29T04:11:31.692 INFO:tasks.ceph:Compressing logs... 2024-03-29T04:11:31.693 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T04:11:31.708 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T04:11:31.732 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48489.log 2024-03-29T04:11:31.733 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T04:11:31.785 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T04:11:31.785 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43997.log 2024-03-29T04:11:31.785 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37530.log 2024-03-29T04:11:31.785 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.3.43997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37410.log 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43541.log 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37530.log.gz 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37290.log 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37410.log: /var/log/ceph/ceph-client.1.43541.log: 90.4% -- replaced with /var/log/ceph/ceph-client.3.43997.log.gz 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37410.log.gz 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37570.log 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37290.log.gz 2024-03-29T04:11:31.786 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.1.43541.log.gz 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37570.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37490.log 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37570.log.gz 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44721.log 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37370.log 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.1.44721.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T04:11:31.787 INFO:teuthology.orchestra.run.smithi203.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37490.log.gz 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37370.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37330.log 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.7.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37370.log.gz 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37450.log 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.37330.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.1.44721.log.gz/var/log/ceph/ceph-client.admin.37450.log: 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37330.log.gz 2024-03-29T04:11:31.788 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37450.log.gz 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mds.d.log: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-29T04:11:31.789 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T04:11:31.855 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48489.log: -- /var/log/ceph/ceph-client.admin.50866.log 2024-03-29T04:11:31.855 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51973.log 2024-03-29T04:11:31.855 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48489.log.gz 2024-03-29T04:11:31.855 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48699.log 2024-03-29T04:11:31.855 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50866.log.gz 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43764.log 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51973.log.gz 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47232.log 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54410.log 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48699.log.gz 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47232.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43764.log.gz 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52839.log 2024-03-29T04:11:31.856 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47232.log.gz 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46124.log 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54410.log.gz 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr:gzip/var/log/ceph/ceph-client.admin.52839.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54149.log 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.46124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49179.log 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52839.log.gz 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51363.log 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54149.log.gz 2024-03-29T04:11:31.857 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45150.log 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.0.46124.log.gz 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49179.log.gz 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52242.log 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45150.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51363.log.gz 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44152.log 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.52242.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45150.log.gz 2024-03-29T04:11:31.858 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48059.log 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43620.log 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48059.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44152.log.gz 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52242.log.gz 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46896.log 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48059.log.gz 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52923.log 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43620.log.gz 2024-03-29T04:11:31.859 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53454.log 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.52923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52176.log 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46896.log.gz 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53454.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52923.log.gz 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53127.log 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53454.log.gz 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52881.log/var/log/ceph/ceph-client.admin.52176.log: 2024-03-29T04:11:31.860 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51513.log 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.52881.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53127.log.gz 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48017.log 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52176.log.gz 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52881.log.gz 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51447.log 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51513.log.gz 2024-03-29T04:11:31.861 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53328.log 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54233.log 88.7% 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48017.log.gz 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45444.log 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53328.log.gz 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51447.log.gz 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50013.log 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51931.log 2024-03-29T04:11:31.862 INFO:teuthology.orchestra.run.smithi156.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54233.log.gz 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50013.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45444.log.gz 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45402.log 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50013.log.gz 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51931.log: /var/log/ceph/ceph-client.admin.45402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47400.log 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51931.log.gz 2024-03-29T04:11:31.863 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37362.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45402.log.gz 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54107.log 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47400.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46786.log 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47400.log.gz 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51153.log 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46786.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54107.log.gz 2024-03-29T04:11:31.864 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48243.log 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51153.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46786.log.gzgzip 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47022.log 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51153.log.gz 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49221.log 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47022.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48243.log.gz 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44764.log 2024-03-29T04:11:31.865 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47022.log.gz 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49137.log 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49221.log.gz 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53911.log 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54539.log 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44764.log.gz 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53911.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49137.log.gz 2024-03-29T04:11:31.866 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46980.log 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53911.log.gz 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48405.log 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44767.log 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54539.log.gz 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46980.log.gz 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52032.log 2024-03-29T04:11:31.867 INFO:teuthology.orchestra.run.smithi156.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48405.log.gz 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50656.log 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.52032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47274.log 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44767.log.gz 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52032.log.gz 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50740.log 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50656.log.gz 2024-03-29T04:11:31.868 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49719.log 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50740.log: gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.47274.log.gz 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr: /var/log/ceph/ceph-client.admin.47316.log 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50740.log.gz 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54368.log 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49719.log.gz 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48824.log 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53085.log 2024-03-29T04:11:31.869 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47316.log.gz 2024-03-29T04:11:31.870 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54368.log.gz 2024-03-29T04:11:31.870 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-03-29T04:11:31.870 INFO:teuthology.orchestra.run.smithi156.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48824.log.gz 2024-03-29T04:11:31.870 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45066.log 2024-03-29T04:11:31.870 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44482.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53085.log.gz -5 2024-03-29T04:11:31.870 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53724.log 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45714.log 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45066.log.gz 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50614.log 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49929.log 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53724.log.gz 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45714.log.gz 2024-03-29T04:11:31.871 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53869.log 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50614.log.gz 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49929.log: /var/log/ceph/ceph-client.admin.47610.log 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48953.log 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49929.log.gz 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53869.log.gz 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53412.log 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47610.log.gz 2024-03-29T04:11:31.872 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47568.log 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48953.log.gz 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53412.log.gz 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50242.log 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48657.log 2024-03-29T04:11:31.873 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53827.log 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47568.log.gz 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50242.log.gz 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48657.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48657.log.gz 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49593.log 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.audit.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53827.log.gz -5 2024-03-29T04:11:31.874 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49761.log 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50097.log 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49593.log.gz 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44480.log 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50097.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49761.log.gz 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46246.log 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T04:11:31.875 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50097.log.gz 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54791.log/var/log/ceph/ceph-client.admin.44480.log: 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47694.log 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44480.log.gz 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46744.log 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46246.log.gz 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47694.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54791.log.gz 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45108.log 2024-03-29T04:11:31.876 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47694.log.gz 2024-03-29T04:11:31.877 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51774.log 2024-03-29T04:11:31.877 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43393.log 2024-03-29T04:11:31.877 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46744.log.gz 2024-03-29T04:11:31.877 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.51774.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45108.log.gz 2024-03-29T04:11:31.877 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43309.log 2024-03-29T04:11:31.877 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47933.log 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43309.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43393.log.gz 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45318.log 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48615.log 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3%/var/log/ceph/ceph-client.admin.47933.log: -- replaced with /var/log/ceph/ceph-client.admin.43309.log.gz 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45318.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47933.log.gz 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2024-03-29T04:11:31.878 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45318.log.gz/var/log/ceph/ceph-client.admin.48615.log: 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49635.log 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48615.log.gz 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47064.log 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2024-03-29T04:11:31.879 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50487.log 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53496.log 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49635.log.gz 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47064.log: /var/log/ceph/ceph-client.admin.50487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45276.log 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47064.log.gz 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50487.log.gz 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48742.log 2024-03-29T04:11:31.880 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53496.log: /var/log/ceph/ceph-client.admin.45276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54318.log 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53496.log.gz 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48742.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45276.log.gz 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44474.log 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48742.log.gz 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50199.log 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54318.log.gz 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53538.log 2024-03-29T04:11:31.881 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50954.log 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44474.log.gz 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53538.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50199.log.gz 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47736.log 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50954.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53538.log.gz 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50954.log.gz 2024-03-29T04:11:31.882 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45024.log 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47736.log.gz 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43059.log: /var/log/ceph/ceph-client.admin.45024.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51093.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45024.log.gz 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42871.log 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2024-03-29T04:11:31.883 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55059.log 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50698.log 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42871.log: /var/log/ceph/ceph-client.admin.55059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45612.log 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42871.log.gz 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55059.log.gz 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54875.log 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53169.log 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50698.log.gz 2024-03-29T04:11:31.884 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49551.log 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54875.log.gz 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44148.log 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45612.log.gz 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49095.log 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49551.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53169.log.gz 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44146.log 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49551.log.gz 2024-03-29T04:11:31.885 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47484.log 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44148.log.gz 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49095.log.gz 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45360.log 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47106.log 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47484.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44146.log.gz 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55101.log 2024-03-29T04:11:31.886 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47484.log.gz 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47106.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.45360.log.gz 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44488.log 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47106.log.gz 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.55101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44861.log 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48363.log 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44488.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55101.log.gz 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43437.log 2024-03-29T04:11:31.887 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44488.log.gz 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5%/var/log/ceph/ceph-client.admin.48363.log: -- replaced with /var/log/ceph/ceph-client.admin.44861.log.gz 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45570.log 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43437.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48363.log.gz 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48531.log 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0%/var/log/ceph/ceph-client.admin.45570.log: -- replaced with /var/log/ceph/ceph-client.admin.43437.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43140.log 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48531.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.45570.log.gz --verbose 2024-03-29T04:11:31.888 INFO:teuthology.orchestra.run.smithi156.stderr: -- /var/log/ceph/ceph-client.admin.47526.log 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48531.log.gz 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49299.log 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47526.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.43140.log.gz -5 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48573.log 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3%/var/log/ceph/ceph-client.admin.49299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54581.log 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47526.log.gz 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48573.log: gzip -5 --verbose -- 89.6% /var/log/ceph/ceph-client.admin.49971.log -- replaced with /var/log/ceph/ceph-client.admin.49299.log.gz 2024-03-29T04:11:31.889 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48573.log.gz 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48159.log 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54581.log.gz 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48159.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49971.log.gz 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50824.log 2024-03-29T04:11:31.890 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54276.log 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48159.log.gz 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50824.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47190.log 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54276.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50824.log.gz 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54276.log.gz 2024-03-29T04:11:31.891 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47442.log 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47190.log.gz 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44145.log 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54749.log 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47442.log.gz 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.52326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2024-03-29T04:11:31.892 INFO:teuthology.orchestra.run.smithi156.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44145.log.gz 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.54749.log: -- /var/log/ceph/ceph-client.admin.48910.log 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44151.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.54749.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.50445.log 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52797.log 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54454.log 2024-03-29T04:11:31.893 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0%/var/log/ceph/ceph-client.admin.52797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49425.log 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48910.log.gz 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50445.log.gz 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52797.log.gz 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43267.log 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49425.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54454.log.gz 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48285.log 2024-03-29T04:11:31.894 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43267.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49425.log.gz 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49887.log 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43267.log.gz 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53953.log 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49887.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48285.log.gz 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47873.log 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49887.log.gz 2024-03-29T04:11:31.895 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53211.log 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53953.log.gz 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37715.log 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53211.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47873.log.gz 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51005.log 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53211.log.gz 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54191.log 2024-03-29T04:11:31.896 INFO:teuthology.orchestra.run.smithi156.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37715.log.gz 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44688.log 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54191.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51005.log.gz 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49845.log 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54191.log.gz 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49053.log 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44688.log.gz 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44154.log 2024-03-29T04:11:31.897 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49053.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49845.log.gz 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47148.log 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49053.log.gz 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53286.log 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44154.log.gz 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47148.log: /var/log/ceph/ceph-client.admin.53286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43806.log 2024-03-29T04:11:31.898 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mgr.y.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53286.log.gz 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55019.log 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47148.log.gz 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54623.log 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.55019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43806.log.gz 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49803.log 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55019.log.gz 2024-03-29T04:11:31.899 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54623.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54623.log.gz 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45486.log 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43182.log 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49803.log: /var/log/ceph/ceph-client.admin.45486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45234.log 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43182.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49803.log.gz 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45486.log.gz 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51321.log 2024-03-29T04:11:31.900 INFO:teuthology.orchestra.run.smithi156.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43182.log.gz 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45234.log: /var/log/ceph/ceph-client.admin.51321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54917.log 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49509.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45234.log.gz 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51321.log.gz 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47778.log 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2024-03-29T04:11:31.901 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53370.log 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54917.log: /var/log/ceph/ceph-client.admin.47778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54496.log 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53370.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54917.log.gz 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47778.log.gz 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44485.log 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53370.log.gz 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43848.log 2024-03-29T04:11:31.902 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54496.log: /var/log/ceph/ceph-client.admin.44485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50139.log 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43848.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54496.log.gz 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44485.log.gz 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50910.log 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43848.log.gz 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43974.log 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50139.log: /var/log/ceph/ceph-client.admin.50910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49341.log 2024-03-29T04:11:31.903 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43974.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50910.log.gz 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50139.log.gz 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43974.log.gz 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48995.log 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44982.log 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49341.log: /var/log/ceph/ceph-client.admin.48995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48201.log 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44982.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49341.log.gz 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48995.log.gz 2024-03-29T04:11:31.904 INFO:teuthology.orchestra.run.smithi156.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44982.log.gz 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50364.log 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44940.log 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48201.log: /var/log/ceph/ceph-client.admin.50364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50529.log 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44940.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50364.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48201.log.gz 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45192.log 2024-03-29T04:11:31.905 INFO:teuthology.orchestra.run.smithi156.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50529.log: /var/log/ceph/ceph-client.admin.45192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.0.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50529.log.gz 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45192.log.gz 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53622.log 2024-03-29T04:11:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43481.log: /var/log/ceph/ceph-client.admin.44766.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53622.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53622.log.gz 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50284.log 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47358.log 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44153.log: /var/log/ceph/ceph-client.admin.50284.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50284.log.gz 2024-03-29T04:11:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53784.log 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49383.log 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47358.log: /var/log/ceph/ceph-client.admin.53784.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47358.log.gz 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53784.log.gz 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51051.log 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49383.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49383.log.gz 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T04:11:31.908 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54833.log 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51051.log: /var/log/ceph/ceph.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51051.log.gz 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43351.log 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54833.log: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54833.log.gz 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50572.log 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48447.log 2024-03-29T04:11:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43351.log: /var/log/ceph/ceph-client.admin.50572.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43351.log.gz 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50572.log.gz 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53664.log 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.48447.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48447.log.gz 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44489.log 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53664.log: /var/log/ceph/ceph-osd.2.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53664.log.gz 2024-03-29T04:11:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53043.log 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44489.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.46622.log 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53043.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53043.log.gz 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44768.log 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.2.46622.log: 90.8% -- replaced with /var/log/ceph/ceph-client.2.46622.log.gz 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44768.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44768.log.gz 2024-03-29T04:11:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43532.log 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43574.log 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43532.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43532.log.gz 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50055.log 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43574.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43574.log.gz 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44486.log 2024-03-29T04:11:31.912 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50055.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50055.log.gz 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43225.log 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44486.log: 91.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54665.log 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44486.log.gz 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43225.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31331.log 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.54665.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43225.log.gz 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54665.log.gz 2024-03-29T04:11:31.913 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53580.log 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46938.log 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.tmp-client.admin.31331.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31331.log.gz 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.53580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51405.log 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53580.log.gz 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50782.log 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51405.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46938.log.gz 2024-03-29T04:11:31.914 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.51405.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.54707.log 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43680.log 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.50782.log: /var/log/ceph/ceph-client.admin.54707.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54707.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50782.log.gz 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48117.log 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48868.log 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43680.log: /var/log/ceph/ceph-client.admin.48117.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43680.log.gz 2024-03-29T04:11:31.915 INFO:teuthology.orchestra.run.smithi156.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48117.log.gz 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44492.log 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44156.log 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44492.log: /var/log/ceph/ceph-client.admin.48868.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48868.log.gz 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44492.log.gz 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52965.log 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54959.log 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.44156.log: /var/log/ceph/ceph-client.admin.52965.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44156.log.gz 2024-03-29T04:11:31.916 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52965.log.gz 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49467.log 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43932.log 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49467.log: /var/log/ceph/ceph-client.admin.54959.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54959.log.gz 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49467.log.gz 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43890.log 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43932.log: /var/log/ceph/ceph-mds.a.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43932.log.gz 2024-03-29T04:11:31.917 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51237.log 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43890.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43890.log.gz 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49677.log 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51237.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51237.log.gz 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43722.log 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.49677.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49677.log.gz 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.43722.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43722.log.gz 2024-03-29T04:11:31.918 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47652.log 2024-03-29T04:11:31.919 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.51195.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2024-03-29T04:11:31.919 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45528.log 2024-03-29T04:11:31.919 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.47652.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47652.log.gz 2024-03-29T04:11:31.919 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.45528.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45528.log.gz 2024-03-29T04:11:31.920 INFO:teuthology.orchestra.run.smithi203.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T04:11:31.920 INFO:teuthology.orchestra.run.smithi203.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T04:11:31.920 INFO:teuthology.orchestra.run.smithi203.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T04:11:31.920 INFO:teuthology.orchestra.run.smithi203.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T04:11:31.921 INFO:teuthology.orchestra.run.smithi203.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T04:11:31.953 INFO:teuthology.orchestra.run.smithi156.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T04:11:31.961 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T04:11:31.964 INFO:teuthology.orchestra.run.smithi156.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T04:11:31.965 INFO:teuthology.orchestra.run.smithi156.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T04:11:31.970 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T04:11:31.972 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T04:11:31.973 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m0.221s 2024-03-29T04:11:31.973 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m0.827s 2024-03-29T04:11:31.973 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.052s 2024-03-29T04:11:31.987 INFO:teuthology.orchestra.run.smithi156.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T04:11:32.101 INFO:teuthology.orchestra.run.smithi156.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-29T04:11:33.143 INFO:teuthology.orchestra.run.smithi156.stderr: 97.2% -- replaced with /var/log/ceph/ceph-client.0.51774.log.gz 2024-03-29T04:11:33.145 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-29T04:11:33.145 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m1.424s 2024-03-29T04:11:33.145 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m2.691s 2024-03-29T04:11:33.145 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.244s 2024-03-29T04:11:33.146 INFO:tasks.ceph:Archiving logs... 2024-03-29T04:11:33.146 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/remote/smithi156/log 2024-03-29T04:11:33.147 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T04:11:33.689 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/remote/smithi203/log 2024-03-29T04:11:33.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T04:11:33.842 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T04:11:33.853 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T04:11:33.853 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T04:11:33.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T04:11:33.945 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T04:11:33.956 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T04:11:33.957 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T04:11:33.960 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T04:11:33.971 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-29T04:11:33.976 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T04:11:33.976 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-29T04:11:33.977 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +74us[ +74us] +/- 79ms 2024-03-29T04:11:33.977 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 0 -1290us[-1290us] +/- 70ms 2024-03-29T04:11:33.977 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv03.front.sepia.ceph.com 3 6 377 1 +1175us[+1177us] +/- 67ms 2024-03-29T04:11:33.977 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:11:33.978 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-29T04:11:33.981 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T04:11:33.981 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-29T04:11:33.982 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +86us[ +86us] +/- 79ms 2024-03-29T04:11:33.982 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 -1279us[-1274us] +/- 70ms 2024-03-29T04:11:33.982 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv03.front.sepia.ceph.com 3 6 377 1 +1223us[+1228us] +/- 67ms 2024-03-29T04:11:33.982 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T04:11:33.983 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T04:11:34.007 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T04:11:34.008 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T04:11:34.044 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T04:11:34.116 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T04:11:34.189 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T04:11:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:11:34.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T04:11:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:11:34.252 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T04:11:34.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T04:11:34.293 INFO:teuthology.task.internal:Duration was 587.095199 seconds 2024-03-29T04:11:34.293 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T04:11:34.323 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T04:11:34.323 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T04:11:34.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T04:11:34.367 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T04:11:34.371 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T04:11:34.576 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T04:11:34.662 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-03-29T04:11:34.662 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T04:11:34.682 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T04:11:34.682 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T04:11:34.711 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T04:11:34.711 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T04:11:34.724 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T04:11:34.780 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T04:11:34.780 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T04:11:34.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T04:11:34.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T04:11:34.932 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T04:11:34.932 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T04:11:34.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T04:11:34.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T04:11:34.998 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T04:11:35.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T04:11:35.026 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-03-29T04:11:35.054 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-29T04:11:35.070 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T04:11:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:11:35.096 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T04:11:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T04:11:35.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T04:11:35.135 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T04:11:35.135 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/remote/smithi156 2024-03-29T04:11:35.136 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T04:11:35.172 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080/remote/smithi203 2024-03-29T04:11:35.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T04:11:35.208 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T04:11:35.209 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T04:11:35.211 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T04:11:35.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T04:11:35.269 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T04:11:35.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T04:11:35.299 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T04:11:35.299 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T04:11:35.302 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T04:11:35.313 INFO:teuthology.orchestra.run.smithi156.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 04:11 /home/ubuntu/cephtest 2024-03-29T04:11:35.316 INFO:teuthology.orchestra.run.smithi203.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 04:11 /home/ubuntu/cephtest 2024-03-29T04:11:35.317 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T04:11:35.334 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T04:11:35.400 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T04:11:35.426 DEBUG:teuthology.nuke:shortname: smithi156 2024-03-29T04:11:35.426 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T04:11:35.451 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-29T04:11:35.452 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T04:11:35.456 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 03:53:10.013994', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvoTFsWWL4vZ7GzTmlpqlvDFCSFJM8MPUM/4cPygSQpM4a+eE5+Ilqcdje7rDFxqSiEeHkRd0vNIKMSl6fIEck='} 2024-03-29T04:11:35.473 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629080', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 03:53:10.012152', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2iheM/phZGpxb/nWh+qNqJPh21pUeEuzfV1bo+S1e/XORiFLwi+n9nvbrRFTnjMjUmEIh7WVJLrqjXkilT/Nk='} 2024-03-29T04:11:35.513 INFO:teuthology.orchestra.console.smithi156:Power off 2024-03-29T04:11:35.513 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T04:11:35.540 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-03-29T04:11:35.540 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:35.558 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-29T04:11:35.559 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T04:11:35.568 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-29T04:11:35.581 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-29T04:11:35.581 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:35.605 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-29T04:11:39.569 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:39.606 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:39.680 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-29T04:11:39.718 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-29T04:11:43.681 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:43.719 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:43.793 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-03-29T04:11:43.830 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-29T04:11:43.894 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-03-29T04:11:47.834 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T04:11:47.945 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-29T04:11:48.046 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-29T04:11:48.151 INFO:teuthology.run:Summary data: description: fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining} duration: 587.0951988697052 failure_reason: 'Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining)' flavor: default owner: scheduled_mchangir@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c5732608d4b84a2da0647cab8b5064b8 status: fail success: false 2024-03-29T04:11:48.151 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T04:11:48.240 INFO:teuthology.run:FAIL